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Introduction 


The  management  guidelines  are  divided  into  three  parts. 
Part  I contains  a number  of  questions  which  site  staff  ought 
to  try  to  answer  when  establishing  or  while  operating  a 
computer-based  education  (CBE)  site.  Each  question  is 
followed  by  a brief  commentary.  Part  II  contains  sugges- 
tions for  selecting  and  training  the  staff  of  a site.  In 
order  to  keep  the  commentary  to  the  questions  in  part  I 
brief,  detailed  discussions  of  some  topics  have  been  placed 
in  part  III  of  these  guidelines.  Part  III  also  contains 
recommendations  and  suggestions  about  topics  not  introduced 
in  parts  I and  II.  For  succinctness,  some  terminology  common 
to  CRF  is  used  without  previously  defining  the  terms. 

The  author  suggests  that  all  administrative  staff 
responsible  for  the  (proposed)  site  read  those  sections  of 
the  guidelines  which  are  relevant  to  their  situation,  return 
to  part  I to  answer  the  questions  for  their  project  or 
organization,  then  compare  their  answers  to  those  of  other 
administrators.  The  commentary  following  most  of  the 
questions  should  help  in  formulating  decisions. 

fl  critical  incident  study  was  completed  as  a prerequi- 
site for  preparing  these  guidelines  (Steinberg,  1977). 
References  to  the  individual  cases  in  that  study  are  given 
in  pointed  brackets,  "<X>". 

The  questions  in  the  following  section  are  arranged 
roughly  from  general  to  specific.  As  can  be  seen  from  the 
table  of  contents,  the  second  half  of  part  I relates  only  to 
sites  wishing  to  use  the  PLATO  system  as  an  instructional 
medium . 


iv 


In  order  to  provide  specific  detailed  advice  and 
examples,  this  document  frequently  refers  to  features  and 
situations  found  on  the  PLATO  IV  system  (Lyman,  1977,  Wood 
1975).  Nevertheless,  most  of  the  suggestions  provided  apply 
equally  well  to  other  CEE  systems  which  offer  the  author 
great  flexibility  in  designing  courseware. 

The  staff  members  of  the  Military  Training  Centers  (MTC) 
Group  and  the  PLATO  Educational  Evaluation  and  Research  (PEER) 
Group,  contributors  to  these  guidelines,  have  gained  much  of 
their  experience  through  working  with  CDE  developments 
within  the  military.  In  addition,  the  staff  have  all 
had  experience  with  civilian,  academic,  or  industrial  CEE 
implementations.  In  general,  we  have  found  classification 
in  terms  of  site  objectives,  cognitive  level  taught,  etc. 
more  useful  than  classification  by  military  and  non- 
military. Therefore,  situations  specific  to  military  users 
are  noted  throughout  the  guidelines  rather  than  in  a 
separate  section. 


PART  I --FUNDAMENTAL  ISSUES  IN  PLANNING  A CBE  SITE 


SITE  OPJFCTi/ES 


To  what  aRencies,  offices,  and  organizations  does  the 
CBE  group  report?  What  objectives  do  these  people 
have  for  CBE? 


This  is  a very  important  consideration  for  the  future 
of  a site,  additional  funding,  etc.  One  cannot  simply  read 
the  charge  or  charter  given  to  the  CBE  staff  and  assume  the 
goals  stated  there  are  complete  or  accurate.  In  fact,  they 
have  often  been  found  to  be  unrealistic  or  hastily-wr i.tten . 
For  example,  one  reason  for  implementing  CBE  may  be  to  gain 
recognition  and  enhanced  public  relations.  Although  this 
goal  would  necessitate  high  public  visibility,  the 
"official"  objectives  might  refer  only  to  educational  uses 
and  purposes. 

In  many  cases  it  is  imperative  as  a first  step  to  bring 
the  organizational  expectations  for  CBE  into  line  with 
reality,  gently,  so  that  no  one  loses  interest  or  respect 
for  the  abilities  and  opportunities  of  the  CBE  system. 


To  what  extent  do  the  various  administrators  and 
agencies  supervising  the  site  agree  on  the  objectives? 


Disagreements  on  philosophies  of  education,  orientation 
towards  operational  training  vs.  research,  etc.  have  seriously 
hampered  efforts  at  several  sites  (Himwich,  1977).  Do  not 
opt  for  a middle-of-the-road  position  just  to  satisfy  people — 
it  is  often  untenable.  For  example,  the  "operational 
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tejc'^ing  vs.  research"  issue  is  basic  and  cannot  be  resolved  | 

sirroly  by  al^'ocatinr  ranpower  within  a single  project  C e . g . , ! 

■ ~ ' I 

two  staff  r.enbers  for  research,  four  staff  members  for 

instruction).  The  di v i ie-and-appease  strategy  may  work  when 

several  projects  are  available,  but  the  constraints  of 

research  are  nearly  always  in  opposition  to  those  for 

teaching . 

Lastly,  don't  assume  that  because  you  have  assembled 
everyone  in  a room.,  gotten  their  consensus,  and  even  signed 
documents  to  that  effect,  that  your  jot  is  now  over.  V'hile 
such  a step  is  important,  it  is  generally  taken  early  in  a 
site's  history,  before  impressions  have  developed.  It  it 
necessary  to  keep  in  touch  with  everyone  (especially  non- 
project  staff)  to  maintain  their  support  and  cooperation. 


Coes  the  implementation  of  CPE  at  the  site  satisfy  a 
single  purpose  (i.e.,  the  fulfillment  of  a project 
commitment)  or  multiple  purposes  (i.e.,  general  sup- 
port of  its  parent  organization)? 


These  guidelines  distinguish  between  single  purpose  and 
m.ultiole  purpose  sites  and  the  sugeesticns  appropriate  to 
each  at  several  points  later  in  the  text.  Typically  a 
single-purpose  or  direct-support  site  has  a definite, 
limited  staff  engaged  in  fulfilling  a research  or  develop- 
ment contract  of  a fixed  length  and  for  a lim.ited  audience, 
in  contrast,  a general-support  site  exists  as  a learning 
resource.  ’’t  must  attract  authors  to  write  lessons,  but  can 
seldom,  pay  them  or  exert  -"uch  cortrol  over  their  product  or 
process.  This  type  of  site  acts  as  a utility,  distributing 
resources  and  services  to  a roughly-defined  group  of  users. 
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Now,  let  us  consider  the  answers  to  the  question. 

Direct-support  only.  Tn  this  case  the  objectives  and 
management  are  relatively  straightforward.  CEE  users  have 
only  a single  objective  and  the  site  has  only  one  way  it  can 
succeed.  For  example,  a course  director  given  two  years, 
a staff  of  four  and  ten  terminals  to  see  if  CBF  could  be  cost- 
effective  in  an  inventory  control  course  would  set  up  a 
direct-support  site. 

General-support  only.  A CDF.  site  o^-ganized  to  provide 
general  support  to  many  small  projects  and  to  a diverse  set 
of  users  and  purposes  is  termed  a general-support  site.  For 
example,  a department  or  school  which  has  20  terminals  and 
hires  some  trainers,  consultants,  and  classroom  proctors  so 
that  its  instructors  can  try  out  CDF  and  use  it  in  their 
classes  v.’ould  set  up  a site  of  this  kind. 

At  a general-support  site,  it  imperative  to  establish 
and  examine  the  objectives  of  the  site  carefully:  CEE 
terminals  have  sometimes  become  a solution  in  search  of  a 
problem.  Converting  general  ideas  about  possible  benefits 
of  CPF  into  a half-dozen  or  fewer  specific  objectives  for  a 
site  is  an  appropriate  way  to  start.  Reviewing  these 
objectives  every  three  or  six  months  can  help  channel  CBF 
resources  into  the  most  promising  developments. 

Both.  If  the  answer  is  "both,"  relative  priorities  must 
be  established  early  to  avoid  conflicts.  General-support 
tasks  can  easily  fall  to  the  project  staff,  excessively  bur- 
dening them  with  unanticipated  and  unwanted  jobs. 


Obj^ectivej  for  a <^en era  1- support  Site 

* Managers  of  direct-support  sites  may  wish  to  read,  but  * 

* not  answer,  the  questions  below.  * 

»«»»«<(»«»»»««»»»»»»»»«»»»«»»»»»»)(«»««««»«»)>»»»»»»)(»»»«»»»»«« 


Do  you  already  have  clients?  V/ill  the  first  task  be 
to  generate  interest? 


Some  people  will  tear  the  keyset  out  of  your  hands  to  get 
to  use  CBE.  T'hey  can  see  its  potential  and  how  they  can 
use  it.  They  are  willing  to  spend  the  time  to  learn  what 
they  need  to  do  to  apply  CRF  in  their  work.  'Rther  people 
are  busy,  cautious,  and  skeptical.  They  have  seen  too  many 
educational  innovations  come  and  go.  Often  after  two  to 
four  years  they  will  accept  and  embrace  CBF — after  they  h? ve 
seen  materials  in  their  field  teach  their  students,  for 
example.  Short-term  projects  and  innovations  are  trouble- 
some company.  Deep-rooted  commitment  develops  slowly. 


What  are  criteria  for  apportioning  terminals,  staff 
time,  etc.  between  types  of  users,  e.g.,  instructors 
vs.  students,  research  vs.  instruction,  organizational 
use  vs.  private  use  (income  tax,  games),  and  groups 
within  one  user  type;  chemistry  students  vs.  biology 
students . 


Example  situa_tion.  An  instructor  or  his  department 
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reserves  time  for  students  to  take  lessons  assigned  for  a 
class.  After  a few  weeks,  half  or  fewer  of  the  students  are 
using  the  CBE  classroom  at  the  reserved  time.  Many  have 
found  it  more  convenient  to  work  late  at  night  during 
unscheduled  times,  others  can  be  presumed  to  be  letting 
their  assignments  "slip,"  and  there  may  be  some  dropouts. 

Problem:  the  CBE  classroom  is  reserved,  but  the  terminals 
are  now  under-utilized.  V/ho  can  use  the  classroom,  under 
what  conditions,  and  how  can  scheduled  students  be 
protected? 

Another  example.  The  psychology  department  requests  use 
of  terminals,  not  for  teaching  or  research,  but  for  access 
to  recreational  lessons.  They  normally  pay  "volunteers"  and 
wish  to  use  CBE  games  as  a reward  for  participation  in  their 
exper  im.ents . Can  they  use  CDF  this  way  or  does  this  create 
a double  standard?  Why  can't  the  history  department 
schedule  recreational  use  for  their  students  and  staff? 


How  focused  and  structured  will  the  project  he?  Will 
it  offer  a service  to  aH  qualified  to  use  CBE?  Can 
students  not  enrolled  in  a course  use  the  computer? 
Can  any  staff  member  use  it  as  s/he  sees  fit  or  will 
only  designated  staff  work  towards  limited  goals  use 
CBE?  Will  CBE  projects  require  approval? 


Mill  on-site  programming  consultation  services  be 
provided?  Can  a user  expect  to  have  programming  done 
by  a programmer? 
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If  you  wish  to  attract  new  users,  or  if  you  can't 
provide  much  released  time  for  potential  authors,  the 
presence  of  a friendly  programmer  may  mean  the  difference 
between  moderate  and  miniscule  lesson  development.  In  fact, 
depending  on  the  characteristics  of  the  expected  lesson 
developers,  it  may  be  unreasonable  to  invest  only  in 
hardware  and/or  computer  services  without  also  including 
funding  for  support  staff  Note:  3R1.4  is  the  number 

of  a case  from  the  critical  incident  list  ( Steinberg ,1977). 


What  amount  of  resources  can  be  expended  towards 
cheap,  but  non-objective-fulfilling  uses? 


A typical  situation:  at  a site  trying  to  improve  the 
training  of  electronics  technicians,  the  instructors  from 
the  medical  group  at  the  school  find  there  are  lessons 
available  which  they  would  like  their  students  to  work 
through.  Other  than  good  will,  the  medical  usage  won't  help 
the  evaluation  of  the  electronics  course.  What  do  you  do? 

Because  of  the  vast  courseware  available  on  the  PLATO 
system,  this  problem  arises  frequently.  Furthermore,  early 
in  a project,  when  few  locally-written  lessons  are  available, 
the  terminals  are  generally  under-utilized,  making  refusal 
difficult.  If  permission  is  given  to  use  terminals, 
instructors  begin  to  depend  on  the  PLATO  lessons  and  are 
indignant  when  they  can  no  longer  use  them  because  the  time 
and  terminals  are  reserved  for  other  lessons  and  students 
<?A3.1>.  Acceding  to  such  requests  consumes  more  than 
terminal  resources.  Typically  disk  space  for  routers, 
courses,  and  comment  files  is  needed.  Because  the  "out- 
siders" are  unfamiliar  with  operating  the  system,  additional 
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time  must  be  spent  setting  up  the  routers  and  teaching  them 
how  to  enter  a class  roster  and  monitor  student  progress. 

Much  good  will  is  earned  and  often  new  and  successful 
projects  get  started  from  such  informal  beginnings,  but  it 
also  happens  that  these  efforts  have  drained  talent  and 
resources  from  a struggling  site.  The  key  is  to  identify 
those  services  which  can  be  provided  with  the  lowest 
comsumption  of  scarce  resources  and  which  will  generate 
good  will  in  the  most  promising  places. 


Objectives  for  a Cbg  Project 


t 

(The  following  questions  related  to  project  goals 
should  be  asked  of  i^ACH  potential  general-support  user.) 


Why  is  CPF  being  implemented? 


The  general  answer  to  this  question  comes  in  two  forms: 
"We  think  we  can  do  'V*  better  if  we  have  CF'F"  or  "We  have 
some  problems  which  we  think  CRF  might  solve."  Poth  are 
reasonable  answers,  but  the  latter  suggests  some  additional 
advantages.  In  order  to  iustify  the  use  of  CPF  in  the 
first  case,  one  must  find  unambiguous  measures  of  "better" 
and  then  appraise  the  value  of  the  improvement.  For  example, 
how  much  is  it  worth  to  increase  the  final  test  score  of 
commercial  pilots  from  P<Ci%  to  90%7  Or  how  much  is  it  worth 
to  decrease  dronouts  by  10%  and  increase  course  attitudes  to 
half  a standard  deviation  above  the  school  average?  There 
is  the  also  the  problem  that  if  cng  is  declared  better, 
someone  else  is  implicitly  determined  to  be  poorer. 

The  questions  above  are  relevant  but  have  a different 
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perspective  whenever  CPF  is  introduced  because  someone  has 
identified  problems  such  as  unacceptably  low  performance  on 
a f'inal  test  or  a high  dropout  rate  related  to  poor  student 
attitudes.  When  C"E  is  introduced  under  these  conditions, 
it  is  probable  that  other  methods  have  been  tried  and  found 
insufficient;  hence  any  solution  is  likely  to  be  welcomed  by 
all.  Furthermore,  the  introduction  of  CDF  terminals  is  less 
Likely  to  be  interpreted  as  a threat  or  an  insult  to  the 
teaching  abilities  of  the  current  staff. 

In  what  domain  do  you  expect  CPF  to  show  an  advantage? 


Reduced  time  and  costs . When  and  if  accomplished,  this 
is  one  of  the  most  easi ly- jus t i f ied  reasons  for  implementing 
CDF.  One  of  the  most  difficult  tasks  in  performing  a compar- 
ison of  costs  for  CPE  and  other  media  is  the  determination 
of  the  costs  for  the  alternatives.  The  costs  for  developing 
and  delivering  materials  via  conventional  media  have  a 
tendency  to  get  "built  into  the  system"  and  hence  may  be 
overlool'ed.  For  example,  photocopying  charges  may  get  moved 
from  the  training  budget  to  the  administrative  budget  or 
costs  for  transporting  students  to  an  instructional  center 
may  come  from  the  travel  budget  rather  than  the  training 
budget.  It  is  possible,  however,  that  a given  CPF  implemen- 
tation may  eliminate  certain  of  these  costs,  thus  enhancing 
its  cost-effectiveness.  To  become  aware  of  such  areas  of 
savings,  one  must  diligently  search  out  all  cost  items  for 
the  conventional  instruction.  Paving  time  (and  thus  money) 
often  means  shortened  training  and  reduced  staff.  Unless 
the  time  and  staff  can  be  used  elsewhere,  there  is  no  real 
saving . 


CPF  costs  arc  also  difficult  to  calculate  since  amortiza 
tion  periods  for  courseware  developrent  and  hardware  are  not 
s t and  ard  i zed  . Furthermore.  v;hen  new  development  is  occurring 
it  is  difficult  to  accurately  apportion  the  effort  directed 
towards  maintenance  so  that  a true  "operational"  cost  can  be 
determined.  Tf  a '"PF.  site  is  new,  it  is  hard  to  estimate  how 
much  of  the  initial  training  and  set-up  costs  are  one-time 
charges  that  should  be  treated  differently  than  other 
recurring  costs. 

Recause  the  measurements  of  costs  and  savings  are 
complicated,  a special  effort  beginning  at  the  inception  of 
the  project,  should  be  made  to  gather,  record,  and  analyze 
cost  data. 

Enhanced  student  performance.  Enhanced  performance 
includes  broader  student  skills,  higher  proficiency  levels, 
or  higher  cognitive  levels.  Measuring  an  improvement  in 
student  performance  requires  a valid  measurement  instrument. 
In  many  cases  this  might  be  a standardized  test  of  achieve- 
ment. In  others,  it  might  be  a criterion  such  as  the  percent 
age  of  students  passing  the  bar  exam,  the  proportion  being 
admitted  to  a college,  medical  or  graduate  school,  or  the 
number  of  mechanics  or  realtors  receiving  certification  on 
their  first  attempt.  In  cases  where  an  accepted  "yardstick" 
is  not  available,  a considerable  effort  may  be  required  to 
generate  and  validate  appropriate  measures. 

Attitudes_^d_rTot  ivaMoji  . Improvement  of  attitude  and 
motivation  are  often  expected  t'^  accompany  innovations.  How- 
ever, unless  poor  attitudes  and  motivation  have  been  observed 
and  blamed  for  other  problems,  "happier"  students  and  staff 
are  not  necessarily  viewed  by  administrators  as  an  asset 
worth  increased  costs. 

Questionnaires  and  rating  forms  are  the  most  popular 
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ways  of  measiirinp;  attitude,  ^ut  tangible  actions  are  probably 
more  convincing  indicators  (e.g.,  the  fraction  of  students 
bakin'?  another  course  in  the  subject,  the  number  choosing 
a career  in  the  field,  the  attendance  record  or  the  voluntary 
dropout  rate ) . 

New  sej^v i.ces , The  opportunity  to  take  advantage  of 
new  services  not  available  previously  is  frequently  cited 
as  a reason  to  acquire  CBF  service.  For  example,  it  may 
be  impossible  to  offer  realistic  simulations  without  a CFE 
facility.  Althourh  this  reason  is  valid,  it  usually  has 
a more  Fundamental  base  in  one  of  the  other  reasons  cited 
above,  e.g.,  to  enhance  student  performance  by  providing 
(simulated)  experience  requiring  higher  cognitive  skills 
such  as  analysis,  synthesis,  and  decision-making. 

Rese^r^.  PLATO  services  provide  tremendotis  opportun- 
ities for  research.  The  availability  of  vast  kinds  and 
quantities  of  student  performance  data  allows  finely-tuned 
educational  research  to  be  carried  out.  "^lie  flexibility  of 
the  computer  and  terminal  allows  investigations  of  hum.an 
psychology  to  be  quickly  set  up,  implem.ented , analyzed,  and 
dismantled,  Rather  than  reworking  hardware  to  adjust  timing, 
images,  or  control  of  the  experiment,  the  experimenter  can 
merely  modify  programming.  The  standard  terminal  peripherals 
are  sufficient  for  many  experiments  and  the  terminal's 
ability  to  control  other  devices  extends  its  flexibility.  By 
putting  all  the  directions  on  the  computer,  the  experimenter 
can  eliminate  subtle  differences  in  the  directions  given  to 
various  groups.  Directions  can  be  precisely  controlled  over 
a long,  period  of  time  despite  changes  in  staff,  etc. 

General  Considerations.  Many  worthwhile  goals  cannot 
be  measured  immediately  (e.p.,  ease  of  updating  knowledge, 
success  in  non-classroom  situations).  Nevertheless,  they 
should  not  be  eliminated  from  the  list  of  reasons  for 
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acquiring  CBE  services. 

A frequently-overlooked  point  is  that  CBE  per  se 
cannot  fulfill  the  objectives  for  its  implementation  listed 
above.  That  is,  simply  writing  and/or  using  CRF  lessons 
will  not  automatically  improve  attitude,  performance, 
retention,  or  offer  the  other  advantages  available.  CBE 
services  offer  a medium  through  which  such  advantages  may 
be  real i zed--not  a guarantee  that  they  will  be  simply 
because  CPE  is  being  used.  If  students  using  CPE  lessons 
have  shown  higher  retention,  it  is  because  teaching  strat- 
egies which  provide  high  retention  have  been  used.  Because 
many  of  these  strategies  require  a CPF  system  such  as  the 
PLATO  system  to  handle  their  complexity,  people  have  erron- 
eously attributed  the  effects  to  the  system  rather  than  to 
the  effects  achievable  v^  the  system.  This  misunderstanding 
has  led  some  sites  and  projects  to  "write  some  lessons  and 
measure  how  they  are  better"--a  path  primed  for  failure. 

Since  cost-effectiveness  always  seems  to  be  an  issue 
of  foremost  importance,  v;e  have  prepared  a list  of  uses 
for  CBE  which  are  potentially  cost-effective.  Naturally, 
many  factors  determine  the  actual  cost-effectiveness  of  any 
endeavor,  but  the  uses  suggested  have  shown  or  suggest  a 
higher-than-aver age  potential.  The  list  is  found  in  part 
III  of  these  guidelines. 


How  will  cost-effectiveness  be  measured? 

This  nuestion  cannot  be  answered  until  the  definition 
for  cost-effectiveness  is  broken  down  and  clarified.  If 
the  site  or  project  must  show  cost-effectiveness,  must  all 
parts  of  it  be  cost-effective?  That  is,  can  some  research 
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or  service  efforts  be  carried  at  a loss  because  of  their 
overall  beneficial  nature?  For  example,  consider  a site 
which  is  nearly  "breaking  even"  and  is  also  supplying  free 
statistical  and  computational  services  to  departmental  and 
agency  staff.  Fven  if  the  users  of  the  statistical  packages 
cannot  be  made  to  pay  for  the  service,  perhaps  the  value  of 
the  service  provided  to  them  can  be  reckoned  and  subtracted 
from  the  expenses  of  the  site.  Tt  also  may  be  practical  to 
show  that  some  groups  or  operations  are  cost-effective, 
though  not  all  are. 

A second  consideration  revolves  around  the  time  period 
for  which  cost-effectiveness  is  calculated.  If  one  includes 
courseware  development  costs,  the  amortization  period  is 
important.  Also  one  must  consider  whether  courseware  devel- 
opment costs  for  CPE  lessons  should  be  measured  against 
courseware  development  costs  for  traditional  media.  If  a 
new  course  has  been  written  to  incorporate  CBF  lessons, 
one  must  measure  cost-effectiveness  in  terms  of  the  costs 
for  replacing  the  CBF  materials  with  conventional  instruction. 

A third  consideration  is  v;hether  cost-effectiveness 
must  be  demonstrated  while  the  terminals  are  under-utilized. 

For  example,  consider  a course  which  saves  $100/student  for 
100  students/year.  If  the  CEE  terminals  are  in  use  1/3  of 
the  time  for  that  course  and  cost  $20 , 000/year , one  has  two 
ways  to  calculate  cost-effectiveness.  Assuming  fewer  terminals 
cannot  service  the  class,  one  can  point  out  that  costs 
exceed  savings  by  $ 1 0 , 000/ year . On  the  other  hand,  extrap- 
olation of  the  costs  and  savings  to  full  utilization  of  all 
terminals  indicates  a savings  of  .♦  10  , OOO/year  . 


V/hat  features  of  the  CPE  system  do  you  intend  to  use' 
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This  question  is  discussed  more  thoroughly  in  part  III, 
"Why  Use  a CFE  System  Like  PLATO?"  That  discussion  suggests 
that  using  only  a single  feature  of  the  PLATO  system  may  not 
be  justified:  less  sophisticated  devices  may  provide  more 
cost-effective  computation  power  or  graphic  terminals  or 
large  networks.  Use  of  a system  like  the  PLATO  system  is 
especially  justifiable  when  several  such  features  are  needed 
together  and/or  when  the  exact  nature  of  the  task  cannot  be 
pre-spec  if ied . For  example,  if  one  knows  the  best  way  to 
teach  bookkeeping,  one  may  be  able  to  choose  a CBE  system 
(or  other  m.edium)  which  just  meets  the  needs  of  the  training 
and  provides  no  extra  capabilities. 


What  is  the  nature  of  the  project  effort? 


This  question  is  important  for  deciding  the  number  of 
terminals,  choosing  the  staff,  and  planning  the  overall 
goals  for  a project. 

Indirectly  instructional.  This  would  include  experi- 
menting with  hardware  and  software;  using  PLATO  notefiles 
and  communications  network  for  surveys,  etc.;  employing 
lessons  and  packages  for  graphic,  electronic,  and  mechanical 
design  or  statistical  routines;  or  studying  CBE  as  a computer 
science  subtopic.  These  uses  require  few  terminals  and 
little  coordination;  only  a small  support  staff  would  be 
needed  in  most  cases. 

Instructional  research . A research  effort  would  require 
moderate  amounts  of  lesson  design,  little  effort  devoted 
toward  establishing  and  operating  a classroom,  and  high 
emphasis  on  collecting  and  analyzing  data.  At  least  one 


person  with  high  programming  proficiency  would  be  required. 
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O[)or:itional  teachinR/traininp.  The  major  emph.sis  here 
wouM  be  toviard  efficient  production  of  high  Quality  mater- 
ials. ^ curriculum  design'^r  and  the  les.son  designers  would 
form  an  important  part  of  the  staff.  L,iai.son  with  students 
and  i n.structor  ;i  reciuire.s  aopronriate  personality  character- 
istic.s  within  the  staff.  The  goals  of  low-maintenance,  easily- 
proctored  lessons  and  st an d a r d i zed  classroom  procedures  dis- 
tinguish this  from  other  types  of  use. 


What  is  the  marnitiide  of  the  project  effort?  Whit  is 
the  total  number  of  hours  of  material  to  he  prepared? 
Vhat  is  the  total  quantity  of  human  resources  available? 


These  are  important  questions  w.hosc  answers  mu  ;t  be 
compared  to  see  if  the  resources  match  the  task.  In  many 
cases  project  planners  have  failed  to  calculate  the  rate 
of  production  required  to  meet  the  stated  poals.  In  other 
cases  thev  have  developed  more  materials  than  the  site's 
terminals  can  deliver.  They  failed  to  calculate  the  avail- 
able terminal  hours  per  week  and  compare  tha*  to  the  class 
size  <?ni.?,  ?D1.3,  ?ni.il>.  Directions  for  appropriate 
planning  calculations  and  obtaining  estimates  of  expected 
values  are  given  in  part  III  of  these  guidelines. 


How  important  is  it  that  the  products  (lessons,  service 
routines,  packages)  from  the  project  be  used  at  other 
sites? 


Although  widespread  usage  is  often  a good,  objective 
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indication  of  the  general  value  of  a series  of  instructional 
materials,  it  is  difficult  to  force.  When  the  materials  are 
well-written  and  effective,  users  may  flock  around  without 
encouragement  from  the  developers.  However,  taking  steps  to 
guarantee  that  materials  will  gain  rapid  acceptance  in 
several  institutions  may  be  counterproductive.  Generally, 
in  order  to  promote  acceptability,  the  proposed  courseware 
is  planned,  outlined,  and  critiqued  by  a committee  of 
proposed  users.  Finding  formats  and  strategies  which  are 
widely  acceptable  may  result  in  wringing  the  "life"  out  of 
the  instruction.  On  the  other  hand,  ignoring  committee 
recommendations  may  limit  the  number  of  users.  In  summary, 
widespread  use  of  lessons  is  a worthwhile  goal  which  may 
sometimes  best  be  pursued  indirectly.  Ff forts  to  produce 
high-quality  lessons  which  are  easy  to  use  and  which  are 
widely  known  through  seminars,  brochures,  etc.  may  be  more 
effective  than  efforts  to  create  a product  acceptable  to  all 
potential  users  and  compatible  with  their  ideas. 

A frequently-ignored  issue  is  that  a project  or  site 
evaluation  is  necessary  whether  or  not  research  questions 
are  being  answered.  Also,  the  evaluation  must  be  planned  as 
the  project  is  being  initiated,  not  as  it  nears  the  end 
<4A2.1,  4A?.3>.  Below  are  examples  of  issues  to  be  negoti- 
ated and  questions  to  be  answered  during  the  initiation  of  a 
project  or  site. 
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SITE/FROJRCT  rVALUATION 

(A  site  manager  readinp  this  section  should  substitute 
the  word  "site"  for  every  occurrence  of  "project".) 


For  the  project  to  survive  and  thrive,  who  MUST  be 
pleased?  Who  else  SHOULD  be  pleased,  if  possible? 

For  each  of  them,  what  will  constitute  a satisfactory 
project? 


The  general  categories  here  are:  the  level  above  the 
project  (sponsor,  supervisor),  the  project  level  (colleague, 
instructor),  and  the  level  served  by  the  project  (client, 
student).  This  question  might  be  restated  as  "What  are  the 
current  problems  of  each  CDF-related  group?"  and  scrutinizing 
those  problems  which  CDF  might  alleviate  or  exacerbate. 

Administrator . In  order  to  fund  CBF,  a president  or 
board  of  directors  or  sponsor  might  have  to  be  shown  one  or 
more  of  the  following  indications  of  "success": 

(a)  cost  savings, 

(b)  total  dropout/fail  rate  cut  in  half  at  a cost  of 
less  than  $500  per  student  retained, 

(c)  overt  evidence  of  strong  acceptance  by  instructors 
(CPF-related  publications,  requests  for  transfers, 
CBF  time  and/or  terminals),  probably  accompanied  by 
media  coverage, 

(d)  "remarkable"  improvements  in  student  attitude, 
attendance,  enrollment  or  performance. 

Instructor.  Most  instructors  would  be  satisfied  if 


their  total  workloads  were  decreased  or  if  their  students 
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showed  heiehtened  interest.  Instructors  would  he  happier  if 
grsdinfj,  test  preparation,  paperwork,  and  remedial  teaching 
were  reduced,  giving  them  more  time  for  classwork.  "^hev  may 
fear  losing  their  job  to  CBE , particularly  when  "cost  savings" 
are  mentioned.  Lastly,  they  want  to  feel  that  they  are  an 
essential  part  of  the  instructional  process. 

Students . Students  would  be  pleased  to  do  "something 
different,"  spend  less  time  on  "busy  work,"  and  work  at 
their  own  rate.  Some  would  like  to  learn  to  program  and 
others  would  like  to  play  rames.  They  don't  want  to  have  to 
learn  another  skill  (using  the  CBF.  system)  in  order  to  learn 
the  course  material.  Unfortunately,  some  may  dislike  the 
thoroughness  and  attentiveness  which  are  required  in  order 
to  complete  a CPF  lesson--they  can't  sleep  in  class.  Students 
enjoy  the  ability  to  make  mistakes  privately,  the  immediate 
feedback  supplied  by  CPF,  and  the  fact  that  they  control  the 
speed  of  presentation. 


Whose  responsibility  is  it  to  see  that  project  goals 
are  realised? 


Experience  has  shown  that  even  when  sites  have  carefully 
considered  and  chosen  objectives  early  in  the  project,  they 
have  sometimes  failed  to  see  them  met  because  (a)  the  objec- 
tives are  never  discussed  with  and  supported  by  the  project 
staff,  (b)  because  progress  towards  meeting  long-term  goals 
is  seldom  checked  by  management,  or  (c)  because  responsibility 
for  meeting  certain  objectives  is  not  assigned  to  anyone. 

An  example  of  the  last  problem  is  that  at  one  site  it  was 
generally  agreed  that  "enlisting,  instructor  support"  for  CPF 
was  critical  to  the  attainment  of  project  success.  The 
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executive  manager  felt  that  the  task  would  be  done  by  the  j 

manager,  the  manager  felt  that  the  the  task  could  best  be  I 

done  by  his  staff,  and  the  staff  felt  that  liaison  was  the 

manager's  role.  The  circular  logic  was  discovered  when  the 

traditional  staff  exhibited  resistance  to  viewing  or  using  ] 

i 

CBE  lessons.  Though  the  CPF  project  had  been  set  up  to  ii 

solve  a widely-recognized  training  problem,  the  traditional  | 

staff  was  unaware  of  the  project’s  goals.  They  perceived  I 

the  CPF  staff  as  an  outside  group  who  thought  they  could  : 

bring  in  a machine  and,  without  working  with  the  current  : 

teaching  staff,  do  a better  job.  I 


Does  the  project  have  a definite  end  or  re-evaluation 
point?  At  what  points  in  the  project  are  evaluations 
of  success/failure  going  to  be  made? 


Failure  to  realize  that  nearly  every  project  does  have 
potential  ending  points  can  be  unfortunate.  Furthermore,  it 
is  essential  to  realize  that  most  go/no-go  decisions  will  be 
made  p,f'j'?Il  lo  the  actual  ending  date  or  contract  period. 

Thjs  the  extra  effort  needed  to  prepare  an  interim  report  of 
successes  and  achievements  may  be  a far  more  effective 
investment  than  effort  devoted  toward  writing  a final  report 
once  all  the  data  has  been  collected  <3B5.1>. 


If  hardware  delivery,  courseware  production,  or  student 
usage  goals  cannot  be  completely  met,  can  a reasonable 
evaluation  be  made? 
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AlthoLiRh  optir^ism  is  important,  it  is  only  prudent  to 
consider  and  prepare  for  potential  problems . In  a sense, 
these  guidelines  are  a detailing  of  potential  problems  for  a 
site  or  project;  the  section  "Contingency  Planninc"  in  part 
IIT  of  these  guidelines  lists  specifically  some  predictable 
problems  and  options.  A report  edited  by  Steinberg  (1b77) 
examines  over  100  incidents  in  the  evolution  of  PLATO  projects. 
Py  learning  about  possible  difficulties,  a site  manager  can 
take  steps  to  protect  and  stabilize  a site. 


Will  the  evaluation  assess  the  attainment  of  a pre- 
established  goal  or  involve  a comparison  with  an 
existing  course  or  program? 


This  is  a basic  question  that  determines  the  character 
of  the  project  and  the  evaluation.  A description  of  the 
pros  and  con s for  each  type  of  evaluation  is  presented  in  a 
chapter  in  part  III.  'briefly,  it  is  extremely  difficult 
to  avoid  comparisons  with  oth<^r  teaching  media  and  techniques 
even  if  the  formal  evaluation  does  not  require  it. 


What  dimensions  will  the  evaluation  examine? 


Piecause  project  objectives  vary  so  widely,  this  section 
can  provide  only  suggestions  about  what  might  be  measured 
and  a few  warnings  about  commonly-overlooked  items.  The 
suggestions  below  should  be  regarded  as  a "shopping  list" 
from  which  to  select  measures  important  for  the  objectives 
of  the  project. 
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Quantity  of  use 

Contact  hours  (The  PLATO  system  automatically 
records  hour s/ter min  a 1 , hnurs/site,  and 
hours/user,  hut  it  doesn't  d istinp.uish 
between  recreational  and  instructional 
lessons . ) 

Number  of  students  or  users 

Number  of  instructional  staff  involved 

Costs 

Production  (Are  traininr  costs  included  in 

development  costs?  If  so,  costs  will  be 
rather  hifth  initially.  If  not,  an  appro- 
priate amortization  rate  must  be  chosen.) 
Oelivery  ( t f development  is  goinp  on  simultaneously, 

it  is  difficult  to  apportion  costs  between 
these  two  activities.) 

General  ('-'hen  considerinr  costs,  add  overhead  of 
retirement,  fringe  benefits,  space,  etc. 

As  noted  elsewhere,  the  manner  of  reporting 
costs  is  important.  In  one  case  the  costs 
for  developing  PLATO  materials  was  times 

higher  than  for  conventional  materials,  but 
the  number  of  users  attracted  to  the  PLATO 
materials  was  9 times  more.) 

Value  of  new  services  (Fven  if  no  one  will  directly 

pay  for  using  the  CPE  system  to  help  do  engineering 
homework  problems,  end-of-course  grade  averaging, 
and  statistical  processing,  the  value  of  those 
services  can  be  estimated.) 
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* if  the  objectives  of  a project  or  site  are  instructional,* 

* some  of  the  follovving  measurements  may  be  relevant.  * 

**********««»*««*»K*«***«*««**«*«*«ft»*ll«*»«*«ft«««»*»»«*«*«ft» 

student  performance  and  attitude 

Consider  using  these:  scores  on  standard  tests, 
class  grades,  attitude  ratings,  pass/fail  ratios, 
absenteeism,  attendence  at  voluntary  sessions, 
lesson  completion  times,  test  completion  times, 
error  rates  in  lessons,  use  of  self-tests  and 
voluntary  review,  extra  practice  problems  done, 
retention  of  training,  number  who  take  follow-up 
courses,  number  in  current  course  who  enrolled 
based  on  recommendation  by  prior  students. 

Quantity  of  lesson  production 

Hours,  on-line  averages  or  replaced  hours  may  be 
measured.  (A  complication:  how  can  routers, 
computer-managed  instruction  (CMI)  routines,  or 
packages  for  circuit  design,  statistics,  etc.  be 
included?  They  have  no  "completion  time.") 
Non-terminal  time  savings  (reduced  homework 
reported  by  students,  reduced  test  preparation  and 
grading  reported  by  staff,  etc.)  may  be  recorded. 

Quality  of  lessons 

At  least  five  different  measures  are  possible: 
use  by  others  (number  of  users,  amount  of  use); 
colleague  opinion  (formally  or  informally  assessed); 
student  opinion  (questionnaires,  on-line  comments, 
the  number  who  take  follow-up  courses);  student 
interaction  data  (error  rates,  proportion  and  type 
of  unanticipated  responses,  requests  for  -HELP-, 
fraction  of  questions  answered  correctly  on  first 
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attempt);  and  the  need  for  proctot  inp,  (number  of 
students  one  proctor  can  handle). 

f'ote:  ’’any  of  the  suppested  evaluation  variables  were 
taken  from  the  October  18,  197^  PLATO  Evaluation  Note 
entitled  "Some  Evaluation  Variables  for  PLATO  Authors,"  by 
P,  Allen  Avner. 

* if  students'  performance  and  attitude  variables  are  to  * 

* be  examined,  the  following  questions  are  relevant.  * 


What  is  the  total  flow  of  students?  How  many  times 
will  classes  start  during  the  evaluation  period? 


The  flow  rate  is  a critical  element  in  determining  the 
structure  of  any  evaluation.  It  imposes  an  upper  limit  on 
the  number  of  students  who  can  be  included  in  the  evaluation 
Because  formative  evaluation  during  lesson  development 
requires  student  "guinea  pigs"  to  test  the  lessons,  the  rate 
at  which  validated  lessons  can  be  produced  is  dependent  upon 
the  frequency  of  new  classes.  Because  evaluation  plans  may 
also  need  revision  and  refinement,  courses  which  frequently 
have  new  classes  forming  are  easier  to  evaluate.  For  exam- 
ple, a one-year  evaluation  period  may  be  sufficient  or  even 
excessive  in  a military  training  situation  where  classes  of 
1 0-?0  students  start  every  two  weeks.  In  a college  or 
university  setting  only  two  or  three  classes  would  start 
during,  this  same  period,  and  a primary  or  secondary  school 
environment  would  provide  only  one  opportunity  to  test  the 
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students.  Mo  matter  how  many  students  are  available,  a one- 
shot  evaluation  is  extremely  hazardous. 

i 


What  is  the  total  impact  on  students  in  term.s  of  num- 
ber of  hours  on  CBE  or  the  total  fraction  of  training 
they  receive  from  CBE?  Is  the  impact  significant 
enough  to  produce  a measurable  effect? 


Giving  half  the  students  in  a class  a four-hour  CBE 
lesson  in  the  middle  of  a 16-week  course  is  unlikely  to 
produce  a measurable  difference  on  the  final  test.  The 
sensitive  and  thorough  data  collection  features  available 
via  CBE  make  many  immediate  measurements  of  effectiveness 
easy,  but  long-term  effects  of  small  or  moderate  amounts  of 
CBE  exposure  may  be  difficult  to  detect. 

In  order  to  perceive  a measurable  effect  on  course 
variables  (dropout  rate,  course  attitude,  etc.)  10%  or  more 
of  the  course  may  have  to  be  taught  via  CBE  lessons.  Fortu- 
nately, individual  chapters,  modules , or  sections  of  a course 
(1-,?  hours)  can  be  successfully  evaluated  if  one  can  examine 
uncontaminated  "micro"  variables.  For  example,  performance 
on  several  test  items  covered  by  the  CPE  instruction  and 
measured  immediately  after  the  instruction  or  within  a week 
or  so  may  show  an  effect  (if  there  is  one).  However,  it  is 
less  likely  that  the  same  variable  measured  at  the  end  of 
the  semester  (when  other  studying  or  sources  of  information 
are  likely  to  supplement  the  student's  knowledge)  will  sho'w 
as  sharp  a difference  between  treated  and  untreated  groups. 


25 


Hysteria  about  being  denied  CPF  training  has  not  reached 
the  level  which  medical  research  must  deal  with,  but  the 
free-flow  classrooms  associated  with  college/university 
settings,  combined  with  full  review  privileges  may  lead  to 
adulterated  samples  when  highly-motivated  students  feel  they 
are  missing  out  on  something.  Conversely,  CPF-trained 
students  who  felt  CPF  training  was  incomplete  have  been 
known  to  study  the  notes  and  texts  of  students  from  control 
groups . 


Will  students  be  compared  to  existing  standards? 


If  existing  standards  are  used,  the  value  of  the  CHE 
lessons  may  not  be  clear.  Because  CPF  lessons  are  likely 
to  he  recently  constructed,  they  may  teach  topics  which 
standardized  tests  do  not  cover  yet;  because  they  may  use 
simulations  and  inquiry  techniques  to  teach  to  a higher  ( 

cognitive  level  than  was  possible  previously,  old  forms  of  i 

I 

tests  may  not  indicate  the  greater  effectiveness  of  the  S 

CBE  materials. 

1 

If  one  creates  new  tests,  how  can  performance  on  two 
different  tests  be  compared?  The  best  answer  to  this 
dilemma  is  to  duplicate  testing.  Continue  to  use  old  test 
forms  for  a while  to  show  (hopefully)  equally  good  results 
with  the  CBE  materials.  Simultaneously  develop  new  tests, 
show  that  they  measure  appropriate  knowledge  or  skills 

(i.e.,  validate  the  tests),  then  measure  the  performance  of  : 

students  receiving  both  traditional  and  CBE  instruction. 

It’s  a lot  of  work,  but  the  only  way  out  of  the  dilemma. 


Are  standardized  end-of-course  tests  available? 


If  so,  and  if  the  tests  in  use  are  valid  for  the  project 
objectives,  you  can  save  a great  deal  of  time  by  not  devel- 
oping new  tests. 


Are  any  tests  giver  upon  entrance  to  the  course? 
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If  ('ritranoe  tests  measure  buckpround  and  aptitude  for 
leurnit-p  the  subject,  vou  may  be  able  to  use  them  to  verify 
the  validity  of  an  evaluation.  For  example,  when  dividing 
classes  for  a comparative  evaluation,  it’s  useful  to  show 
that  both  classes  are  equivalent.  .Puch  measures  may  also  be 
useful  for  staiistical  control  or  for  assessment  of  af.titude- 
t real  men t interactions. 


Will  the  evaluation  assess  the 
The  staff?  The  training?  The 

project  itself? 

management? 

Who  will  carry  out  evaluation; 
outside  group? 

project  staff  or  an 
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PROJECT  PLAMHinC 


What  proportion  of  the  total  available  terminal 
hours/week  are  being  set  aside  for  project  use? 


This  question  is  designed  to  prevent  overschedul i ng  of 
terminal  resources.  In  ordinary  circumstances,  one  finds 
that  terminals  are  used  about  30-40  hours  per  week.  At 
open-use  sites  such  as  universities  an  additional  20-30 
hours  per  week  are  logged  during  night  and  weekend  periods. 
Some  interpretation  of  these  numbers  is  needed.  One  can  and 
should  schedule  more  than  30-40  hours/terminal/week,  but  one 
should  not  expect  that  actual  terminal  use  will  be  100%  of 
what  is  scheduled.  Students  who  finish,  early,  students  who 
are  absent,  and  classes  that  don’t  use  all  available  termi- 
nals contribute  to  a "shrinkage"  of  the  utilization  rate. 

The  20-30  additional  hours  cannot  ordinarily  be  scheduled 
because  students  may  be  unwilling  to  take  the  risks  of  lost 
data  or  uncertain  computer  availability. 


What  is  the  basic  orientation  of  the  staff  in  charge 
of  the  CBE  site? 


Three  types  seem  to  dominate:  computer  scientists, 
educational  specialists,  and  departmental  staff.  They  each 
have  strengths  and  weaknesses. 

Computer  scientists.  They  master  the  system  quickly 
and  they  may  be  able  to  perform  maintenance,  but  some  of 
them  have  a predilection  for  disassembling  equipment  and/or 
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connectinp;  additional]  equipment,  either  of  which  makes 
use  for  ordinary  purposes  more  difficult. 

Fducaticn  specialists.  Their  abilities  with  instructional 
design  may  be  a significant  help  in  producing  high  quality 
courseware,  but  their  concerns  for  instructional  design  may 
lead  them  to  decide  that  only  experienced  designers  should 
be  allowed  to  write  lessons.  In  some  cases  their  zeal  for 
evaluation  will  produce  so  many  surveys  and  questionnaires 
that  potential  users  will  dwindle. 

Departmental  staff.  When  a department  acquires  CBE 
terminals  there  is  sometimes  a tendency  towards  unreasonable 
possessiveness;  no  matter  how  under-utilized  the  terminals 
are,  no  one  but  students  or  staff  from  that  department  is 
allowed  to  use  them. 


At  the  end  of  the  project,  who  will  own  or  control  the 
eouipment:  the  sponsor,  the  principal  investigator,  the 
department  or  group  funded,  or  the  parent  organization? 


This  is  one  item  that  should  be  answered  by  a careful 
reading  of  the  contract  rather  than  by  requesting  an  oral  or 
written  opinion  from  an  administrator. 
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FINANCIAL  CONSinFr./'TIOMS 


Will  the  terminals  and  computer  equipment  be  insured? 


Items  should  be  inventoried  so  that  insurance  policies 
or  other  protection  will  apply  to  stolen  or  vandalised 
equipment . 


Will  a department,  apency  or  rrour  which  uses  CPE  be 
rewarded  in  their  budpet  for  be’.np  innovative  or  charged 
for  using  an  expensive  resource  to  liphten  their  load? 


Both  of  the  views  above  seem  logical:  a department 
should  not  be  penalized  for  taking  advantace  of  CBE,  but, 
conversely,  they  should  be  charged  in  some  way  if  teaching 
burdens  are  lightened.  Perhaps  the  only  benefits  are  to  the 
students.  In  any  case,  justifications  for  such  administrative 
decisions  are  subject  to  criticism  from  either  side  and 
should  thus  be  considered  in  advance. 


V/ho  will  own  lesson  copyrights  and  collect  royalties? 


Funding  Agencv  — It  may  want  them  to  be  free  for  all 
users,  but  it  can't  provide  maintenance  or 
updated  versions. 

Parent  Institution--It  provides  office  space,  possibly 
terminals  or  more. 
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Department--! t may  provide  terminals,  released  time, 
and  students. 

Author--De  may  claim  authoring  was  done  in  "free"  or 
"research"  time  (in  academic  circles,  this  is 
similar  to  acting  as  a textbook  author). 


Have  you  prepared  a complete  budget? 


The  list  below  includes  some  frequently-overlooked 
expenses  incurred  by  CPF  sites. 

Long  distance  telephone  calls  (Troubleshooting 

terminal  problems  is  normally  attempted  this  way. 
The  large  distances  between  most  CPF  sites 
increases  normal  telephone  usage.) 

Staff  acquisition  (Interview  trips,  etc.) 

Conferences  (Since  exchange  of  information  is  vital  in 
this  rapidly  changing  field,  staff  members  must 
travel  to  conferences.) 

Consultation  (Experienced  CPF.  staff  are  hard  to  find. 

A site  may  have  to  engage  consultants  to  provide 
certain  skills  and  services.) 

Hew  journals  and  library  acquisitions 

Postage  and  shipping  for  repair  parts  and  repaired 
terminals 

Above  average  copier  usage 

Room  preparation  (Lighting,  sound  absorption,  air 
compressors  and  pipe,  extra  electrical  power) 

Furniture  (Terminal  carrels,  chairs,  study  tables) 

Printer  supplies  (Paper,  ribbons  or  ink) 

Supplies  for  peripherals  (Microfiche,  audio  disks, 

illustrator  fees,  film,  developing,  tape  recorders) 
See  Francis  (1976)  for  estimates. 
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ENVIPONMFMT  AMD  PHYSICAL  PLANT 


Where  should  the  terrrinals  he  located? 


Unfortunately,  some  laboratory  environments  are  too 
hot  (over  80  degrees  F),  too  dusty,  or  contain  corrosive 
atmospheres.  Some  sites  have  installed  telephone  lines  to 
small  classrooms,  demonstration  rooms,  and  offices  so  that 
a few  terminals  mounted  on  carts  can  be  shared  among  infre- 
quent users.  In  ]ighter  moments,  this  is  referred  to  as 
"terminal  a la  cart." 


What  is  the  size  of  your  proposed  CDF  classroom? 


Pe  sure  to  include  room  for  carrels,  communication 
equipment,  audio  units  or  other  peripherals,  air  compressor 
for  microfiche,  microfiche  library  area,  proctor  location, 
and  possibly  a study  area  for  off-line  activities. 


What  is  the  distance  to  the  CPF  classroom  from  other 
normally  frequented  areas  (housing,  food  centers,  other 
classrooms,  or  libraries)? 


.“Surprisingly,  one  of  the  strongest  consistent  dislikes 
college  and  university  students  have  expressed  about  PLATO  is 


the  distance  to  the  terminals.  Several  otherwise-promising 
military  uses  were  discarded  when  it  was  realized  that  the 
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time  savings  achieved  by  PLATO  instruction  would  be  offset 
by  the  time  needed  to  get  to  and  from  the  PLATO  classroom. 

On  the  other  hand,  usage  statistics  suggest  that  scattered, 
small  classrooms  (6  to  8 terminals)  are  often  inefficient 
in  utilizing  their  terminals. 

What  physical  arrangements  are  planned:  a learning 
resource  center,  a media  center  or  CBE  classroom? 

Will  there  be  separate  locations  for  authors? 

Will  authors  use  terminals  in  a separate  room,  or 
have  reserved  terminals?  Where  will  demos  be  held? 

Will  individuals  have  terminals  in  their  offices? 

Terminals  in  offices  often  have  low  use.  An  authoring/ 
demonstration  room  may  be  advisable  so  that  students  are  not 
disturbed  by  visitors.  Install  only  a few  terminals  so  that 
all  are  nearly  always  occupied  and  authors  must  overflow  into 
the  student  classroom.  Because  it  is  more  likely  that  authors 
will  use  "student"  terminals  than  that  students  will  use 
"author"  terminals,  total  terminal  utilization  will  be  maxi- 
mized with  this  arrangement.  A further  benefit  is  that  authors 
will  be  able  to  observe  v/hat's  happening  in  the  classroom. 

Will  pi  ./ject  staff  members  have  their  own  offices? 

Where  in  relationship  to  those  of  other  staff? 

It  is  often  valuable  to  give  staff  with  part-time  CPF 
appointments  a second  office  near  the  CPF.  installation  in 
order  to  encourage  them  to  devote  a fair  portion  of  their 
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tirr.e  to  CBE  activities.  Tt  is  a rnistake  to  establish  a CRE 
office  area  away  from  the  offices  of  subject  matter  experts 
and  instructors  working  in  related  topics.  This  tends  to 
promote  polarization  and  to  increase  liaison  oroblems. 


How  many  hours  a day  will  terminal  access  be  permitted? 
Who  will  maintain  a schedule?  Who  will  proctor  the 
terminals?  How  will  access  to  student  and  author  areas 
be  controlled? 


Ihe  answers  depend  on  the  type  of  site  and  the  type  of 
use,  but  several  points  seem  to  find  general  agreement:  (a) 
it  is  inefficient  to  employ  authors  as  proctors,  although  in 
small  sites  (fewer  than  8 terminals),  it  may  be  necessary. 

Good  authors  have  a talent  which  is  too  rare  and  costly  to 
warrant  interrupting  their  production  with  details  which 
could  be  handled  by  a proctor  being  paid  a lesser  wage:  (b)  it 
is  inefficient  to  lock  up  the  terminals  after  R:00  p.m.  if 
there  are  potential  users  for  evening  hours:  (c)  unattended, 
unproctored  terminals  quickly  accumulate  food  and  beverage 
debris  and  have  become  targets  for  vandalism  in  a few  cases. 


Will  study  materials,  tests,  microfiche,  etc.  need  to 
be  checked  out  from  a proctor  or  library? 


These  needs  are  additional  considerations  which  may 
influence  the  location  and  layout  of  a PLATO  site.  8inc( 
many  sites  have  only  microfiche  needs,  several  special 
arrangements  have  been  found  to  be  worKabie,  .>ome  sites 
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which  service  only  a limited  user  population  (which  needs 
only  a few  microfiche)  seal  the  slide  selector  door  from  the 
inside  so  that  the  microfiche  can  be  permanently  left  in  the 
terminal.  When  several  different  microfiche  are  needed  at 
one  site,  lessons  may  be  designated  to  work  only  at  certain 
terminals.  .Some  sites  check  out  microfiche  from  a nearby 
library.  Both  kinds  of  sites  have  found  that  theft- 
prevention  procedures  are  needed. 

'i 


1 

1 
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STAFF  SFLECTIOM 

Staf£in^ a Direct-Support  (Project)  Site 

V/ill  the  staff  be  recruited  from  inside  or  outside  the 
ori^anization , department,  course,  etc.?  Can  new  staff 
be  hired,  or  must  existing  staff  be  reari anged? 

In  order  to  acquire  staff  experienced  in  CBE,  one  .i 

i! 

generally  needs  to  look  outside  the  organization.  t'owever, 

if  the  staff  contains  too  large  a fraction  of  new  people,  ■ 

the  project  may  be  considered  an  outsider's  venture.  Some 

l' 

existing  staff  members  with  numerous,  broad-based  contacts 

(not  necessarily  "old-timers")  should  be  included  for  ] 

purposes  of  liaison,  especially  in  a project  whose  goals  ' 

include  operational  teaching  and  training  <2C3.1,  2C3.P>. 

i; 

■; 

Will  the  CBE  project  have  a separate  staff  or  will  : 

most  or  all  of  the  staff  hold  other/previous  department  i; 

affiliations  and  supervisors? 

j 

There  is  a potential  problem  "collecting"  the  released 
time  promised  by  departments--especially  when  the  fraction  ^ 

i 

of  released  time  is  25%  or  less.  Also,  if  control  of 
assignment,  recommendations  for  promotions,  etc.  is  retained 
by  another  department,  there  is  an  authority  vaccuum  for  the 
director  of  the  CBE  project.  Unless  the  "home"  departments 
are  strongly  committed  to  CBE,  staff  members  may  be  rewarded 
for  ignoring  their  CBE  duties  in  order  to  participate  in 


departmental  committees,  tasks,  and  functions.  Hn  the  other 
hand,  if  it  is  necessary  to  fund  a CPF  project  with  ’’soft" 
money,  it  may  he  possible  to  lure  talented  staff  only  by 
giving  them  status  and  a position  with  an  established,  on- 
coinr  department. 


Will  authors  work  full  time  or  will  they  divide  their 
time  with  other  teaching,  operating,  or  research  tasks? 


This  topic  is  discussed  in  greater  length  in  part  II, 
"Choosing  a staff."  Briefly,  full  time  may  be  too  much  for 
some  positions,  such  as  authoring,  but  as  noted  elsewhere, 
structured  tasks  such  as  teaching  tend  to  get  more  than  their 
proportionate  amount  of  effort  compared  to  unstructured 
tasks  such  as  authoring  CPF  lessons.  <?Cd.1,  ?Ci\.2,  PC't.3, 
PC5.1,  ?C'3.2,  2CP.3> 

For  example,  one  half-time  author  who  taught  in  higher 
education  attempted  to  prepare  lessons  in  the  morning  and 
teach  classes  in  the  afternoon.  he  found  that  his  mornings 
were  punctuated  by  students  dropping  in,  meetings  with 
course  staff,  and  ’’rush’’  jobs  from  his  teaching  tasks: 
writing  tests,  grading  lab  books,  etc.  In  contrast,  few 
PLATO  requirements  were  so  major  as  to  cause  him  to  miss  a 
lecture  or  not  supervise  a lab  session.  He  summarized: 

"If  you  don’t  have  a lecture  ready  or  a test  graded,  you'll 
get  people  very  upset.  If  the  PLATO  lesson  isn’t  finished, 
you  just  give  a lecture  instead.  No  one  gets  upset.  Mo 
one  can  say  how  long  it  should  take  to  write  a CBE  lesson." 


Will  authors  work  independently  or  as  part  of  a team? 
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This  question  is  discussed  in  part  II  as  well  as  in 
Mahler,  Misselt,  Schell,  & Aldernan  (197^!).  Briefly,  team 
structures  are  advised  v;hen  a rapid  start-up  is  needed  or 
when  it  is  not  practical  to  teach  the  subject  matter  experts 
to  become  CBF  authors  <3C3.3,  30^.1,  3C4.2,  3CJ(.3,  3Cn.^>. 


Will  everyone  who  is  trained  in  CDE  authoring  be 
expected  to  become  part  of  the  project? 


The  discussion  in  part  II  suggests  that  if  possible,  a 
non-pre judicial  escape  option  be  provided  so  that  people 
who  are  not  removed  by  early  screening  procedures  can  later 
be  moved  to  and  from  other  projects,  tasks,  and  assignm.ents 
within  and  outside  of  the  CPF  nroject.  This  requires  that 
job  descriptions  and  staff  expectations  be  prepared  with 
such  moves  in  mind.  Frequent  within-pro ject  re-assignments 
have  been:  an  author  who  had  difficulties  with  TUTOR  or 
with  instructional  design  becomes  a subject  matter  expert; 
an  author  with  exceptional  skills  becomesta  reviewer  and, 
later,  an  instructional  designer;  a programmer  with  inherent 
instructional  design  skills  becomes  an  author;  and  colleague 
reviewers  who  have  learned  TUTOR  becom.e  authors  <?C3.1>. 

ii 


Will  promotions  change  job  descriptions  and  force 
staff  to  leave  or  upset  the  project  organization? 


Particularly  in  the  military  services,  but  to  a degree 
in  civilian  employment,  promotions  may  disrupt  the  orderly 
progress  of  a project  by  forcing  new  leadership  roles  on 
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the  promoted  group  member  or  by  forcing  these  workers  to 
positions  outside  the  project.  The  potential  for  this 
problem  should  be  analyzed  as  part  of  staff  selection  and 
contingency  plans  should  be  prepared  <2C2.3,  3A3.1>. 

Staffing  a Pe ner al-Suppor t Site  (Volunteer  Authors) 

This  kind  of  site  is  sometimes  found  in  conjunction 
with  a project  (or  direct-support)  site  and  often  occurs  in 
public  higher  education  institutions.  Typical  examples 
would  include  universities  or  medical  schools  where  the 
authors  are  professors  or  instructors  who  wish  to  create 
courseware  for  their  own  courses.  They  might  wish  to  draw 
upon  the  talents  of  a programmer  pool  and  to  consult  with  an 
instructional  designer.  The  degree  to  which  each  would 
learn  CPF  skills  would  vary,  but  many  would  not  wish  to  try 
to  become  experts  in  yet  another  field. 


Who  is  available  to  be  an  author?  What  are  the 
practical  limits  and  what  rewards  are  available?  Must 
existing  tasks  be  rearranged  or  may  new  staff  be  added? 


Some  agencies  have  erroneously  believed  that  merely 
providing  access  to  CPF  services  is  sufficient.  There  may 
be  false  economy  in  not  providing  a large  enough  support 
staff  to  aid  and  direct  appropriate  use  of  the  facilities 
<3P1.4,  3h1.5>. 


Can  a university  author  count  lessons  he  has  prepared 
as  "publications.”? 


AO 


Up  to  now  the  answer  has  generally  been  "no,"  but  this 
issue  may  be  negotiable  and  feelings  about  it  have  been 
changing.  The  main  stumbling  block  has  been  the  absence  of 
"jurying"  or  a "review"  comparable  to  that  a publication 
in  a professional  journal  would  receive.  With  or  without 
a substitute  for  this  "review"  it  is  worth  trying  to  get 
courseware  counted  towards  publications  since  this  issue 
could  be  pivotal  to  a professor  already  pressured  for  time. 
At  least  one  project  has  hired  respected  authorities  to 
critique  their  CBE  production  and  has  used  the  critiques 
as  support  for  their  arguments  to  give  publication  credit 
for  CPE  courseware. 
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AUTHOR  TPflIHING 


Where  will  new  authors  be  trained? 


Tf  a site  has  been  able  to  hire  staff  members  of  suffi- 
cient background  and  experience  co  train  the  rest  of  the 
group  in  the  CEE  language,  instructional  design,  and  using 
the  CPE  system,  the  answer  to  this  question  is  obviously 
"on-site."  Pecause  experienced  staff  are  rare,  many  sites 
are  forced  to  rely  upon  outside  instructors.  The  advantages 
to  training  one's  staff  off-site  seem  to  slightly  outweigh 
on-site  training. 

On-site . If  the  trainers  can  be  brought  to  the  site, 
costs  are  typically  lower  because  fewer  people  must  travel 
and  additional  staff  can  be  trained  or  can  consult  with 
experienced  users  for  only  the  cost  of  their  time.  Though 
the  advantages  of  being  trained  by  several  different  trainers 
is  lost,  it  may  be  compensated  for  by  a longer  training 
period  (made  possible  by  lowered  costs). 

Off-site . Training  at  a large  CPE  center  proyides 
trainees  with  much  broader  experiences  than  they  could  receive 
while  staying  at  home.  Important  experiences  include: 
visiting  a functioning  classroom,  meeting  other  authors  and 
subject  matter  experts  in  the  same  field,  arranging  for 
reviewers  and  users  for  future  materials,  and  viewing  or 
using  optional  devices  available  such  as  terminal  peripherals. 

If  many  staff  are  to  receive  CBF  training,  but  only  a 
few  terminals  are  to  be  installed  at  the  new  site,  there  may 
be  an  advantage  to  obtaining  training  at  a large  CRE  center 
where  terminals  can  be  reserved  for  training  classes. 
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Can  the  staff  pet  credit  for  the  CPE  training? 


It  is  important  for  staff  in  the  military  service,  as 
well  as  many  primary,  secondary,  and  community  college 
staff  members  to  receive  written  certificates  documenting  the 
satisfactory  completion  of  training. 


How  can  the  project  staff  be  protected  from  the  normal 
jealousy  associated  with  special  projects  and  advanced 
equipment? 


Many  of  the  keys  to  preventing  jealousy  and  maintain- 
ing cooperation  have  been  mentioned  previously:  avoid 
hoarding  equipment  and  terminal  time,  include  part-time  as 
well  as  full-time  staff,  indicate  with  words  and  actions  the 
important  role  of  non-project  staff  to  the  overall  project, 
demonstrate  regularly  and  fervently  the  common  goals  being 
addressed  by  both  project  and  non-project  staffs. 

The  need  for  liaison  to  promote  good  will  between  CPF 
and  non-CBE  groups  is  nearly  always  underestimated.  The 
introduction  of  a new  technology  which  is  not  familiar  to 
all  the  staff  is  rather  threatening.  The  possibility  of 
getting  left  out  or  left  behind  is  clear.  Efforts  should  be 
made  from  the  beginning  of  any  project  to  orient  and  intro- 
duce staff  to  the  CBE  system.  The  construction  of  the  CBE 
classroom  should  make  it  easy  for  non-project  staff  to 
observe  what  is  happening  and  use  the  terminals  themselves 
without  feeling  they  are  intruding  <2A1.1,  2A1.2,  2A1.3, 
2A1.'<,  2A2.2,  2A2.3,  2A2.i4>. 
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* The  remaininf:  questions  in  part  I of  these  guidelines  * 

* refer  to  situations  in  which  the  CPF:  system  is  to  he  * 

* used  primarily  for  instruction.  * 


COURSE  DFSCRIPTIOM  AND  ISSUES 

Throughout  the  next  section  it  is  important  that  the 
concept  of  a "course"  is  clear.  '.,'e  will  use  it  to  mean  a 
presentation  of  multiple  concepts  during  a period  of  a month 
or  more.  Single-concept,  short  term  training  presentations 
will  be  refered  to  as  "mini-courses." 


Is  course  lock  step,  tjroup  paced,  or  self  paced? 
Will  this  format  continue  to  be  used? 


Lock  step.  This  format  allows  for  very  easy  scheduling 
of  lessons  and  very  high  terminal  utilization  rates.  In 
higher  education  environments,  however,  students  will  use 
unscheduled  times  and  terminal  locations,  if  available. 

This  tendency  lowers  the  utilization  rate  during  scheduled 
times,  but  does  not  affect  overall  usage  rates.  Courseware 
in  a lock-step  format  is  easy  to  introduce  into  a course: 
the  instructor  simply  informs  the  students  that  they  should 
report  to  the  CRE  classroom  rather  than  the  lecture  hall. 
Because  most  students  will  be  taking  the  same  set  of  lessons 
during  a class  period,  computer  storage  (ECS)  is  efficiently 
used,  and  if  the  lessons  are  well-polished,  little  proctor- 
ing  is  needed.  Although,  by  definition,  no  time  can  be  saved 
with  lock-step  usage,  the  intrinsic  self-pacing  of  CPE  will 
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normally  force  the  instructor  to  make  some  provision  for 
slower  students  to  finish  during  a period  outside  normal 
class  hours.  It  may  also  he  necessary  to  provide  supple- 
mentary materials  for  those  who  finish  early. 

Group  paced,  Structured  study.  "Structured  study  is  a 
situation  in  which  all  students  in  a class  spend  the  same 
fixed  amount  of  time  studying.  This  situation  may  be  brought 
about  by  limited  terminal  access  time  or  by  the  limited 
training  day  found  in  some  military  training  environments. 

In  a structured-study  situation  CPE  strategies  involving 
forward  branching  or  pre-testing  to  skip  sections  of  lessons 
and  thus  accelerate  the  progress  of  bright  or  experienced 
students  are  found  to  be  self-defeating.  They  tend  to  put 
inordinate  pressures  on  slow  students  because  the  self- 
paced,  individualizing  nature  of  CBE  separates  students  in 
terms  of  total  time  to  master  the  topic.  FJence  a lesson 
that  is  judged  or  proven  to  be  efficient  under  self-paced 
conditions  may  be  inappropriate  and  even  inefficient  under 
group-paced  conditions.  Tt  is  especially  true  that  certain 
strategies  "normally"  used  in  CPE  lessons  will  increase 
lesson  production  time  without  saving  student  time.  This 
occurs  because  the  rate  of  the  slowest  student  determines 
the  rate  of  the  group. 

In  both  lock-step  and  group-paced  formats,  students 
finishing  early  can  create  problems.  They  must  be  released 
from  class  or  given  recreational  lessons  or  lessons  on 
optional  topics.  (Giving  them  lessons  used  later  in  the 
course  would  only  put  them  farther  ahead).  Slow  students 
may  have  to  retake  a lesson  in  an  atmosphere  of  shuffling 
chairs  and/or  "pinball  alley"  (i.e.,  while  other  students 
are  loudly  enjoying  r jcreat ional  lessons).  As  the  slow 
students  struggle  to  finish,  they  realize  that  the  progress 
of  the  whole  class  is  being  held  up  until  they  finish.  In 


45 


such  a situation  both  fast  and  slow  students  save  time  by 
"exchansinp  answers"  to  the  questions  in  the  lesson. 

Croup-pacing  is  often  used  when  the  nature  of  parts  of 
the  instruction  is  such  that  self-pacing  is  not  possible,  e.p., 
demonstrations,  close  laboratory  supervision,  etc.  It  is 
often  seen  as  a compromise  between  self-pacing  and  lock-step, 
because  of  the  existence  of  the  lock-step  phases  in  the 
training,  the  number  of  instructors  is  generally  the  same 
as  that  for  lock-step  training.  rcc  demands  on  PLATO  are 
only  slightly  larger  than  that  for  lock-step. 

respite  the  above  problems,  group-pacing  can  reduce  the 
training  oeriod  required  while  adapting  to  varying  student 
populations.  The  latter  is  an  important  attribute  for  the 
military  services  whose  students  may  vary  in  experience  with 
the  time  of  year  or  the  existence  of  a draft  or  mobilization 
ef f ort . 

Group  paced.  Free  study.  Tn  this  situation  CPE  could 
be  used  freely  outside  class,  but  the  class  would  still  be 
group-paced.  Unfortunately,  our  experience  does  not  include 
any  examples  of  this  training  format.  The  most  typical^ 
free-study  environment,  public  education,  supplies  few,  'if 
any,  payoffs  ^'or  early  completion.  Because  presentations 
and  demonstrations  typically  require  the  orderly  completion 
of  prerequisite  materials,  this  format  is  of  limited  utility. 

.Self  paced.  .Self-pacing  is  ideally  matched  to  a CBE 
environment,  but  it  requires  stand-alone  equipment  setups 
or  additional  instructor/proctors  in  cases  where  laboratory 
practice  accompanies  the  course.  Because  students  in  a 
self-paced  environment  often  use  CBE  at  random  times  and 
places,  an  instructor  is  typically  unavailable.  In  most 
PLATO  CBE  situations,  this  presents  few  problems,  since 
notes  between  students  and  their  instructors  are  easily 
written  . 
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without  firm  guidelines,  students  tend  to  allow  self- 
paced  coursework  to  be  displaced  by  assignments  with  definite 
deadlines.  Self-pacing  provides  both  the  problem  and  the 
solution  here.  Tn  some  institutions,  such  courses  can  simply 
be  finished  the  next  term.  Alternatively,  the  instructor 
can  set  up  intermediate  milestones  with  deadlines  that  must 
be  met  by  self-paced  students.  Because  self-pacing  suggests 
that  computer-generated  or  computer-selected  quiz  and  test 
questions  are  needed  (for  test  integrity),  production  of  a 
self-paced  course  may  take  longer  than  for  a group-paced  or 
lock-step  course.  However,  administrative  problems  associated 
with  remediation,  absences,  etc.  can  be  nearly  eliminated. 


What  are  the  roles  of  instructor  in  each  of  the 
training  formats? 


Lock  step.  Because  this  is  the  familiar,  conventional 
teaching  format,  it  will  be  used  as  a basis  for  comparison 
but  will  not  be  discussed  explicitly. 

froup  paced, Structured  study.  An  experienced  military 

curriculum  developer  argues  convincingly  that  thr's  mode  of 
operation  is  the  worst  possible  for  instructors.  Because 
the  slowest  student  sets  the  rate,  the  instructor  cannot  sit 
down  to  review  and  carefully  teach  the  slow  student--that 
would  mean  the  group  would  move  at  a still-slower  rate.  All 
the  instructor  can  do  is  to  prod  the  student,  or  sit  at  the 
terminal  and  give  him  the  answers  so  the  class  can  proceed. 

Sel f paced . In  this  mode,  the  instructor  can  stimulate 
fast  students  and  help  slow  students,  while  the  computer 
carries  the  burden  for  most  information  presentation  and 
testing.  The  computer's  dominant  role  may  bring  out  the 
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best  in  dedicated  staff,  the  worst  in  lazy  staff  ("If  CPF  is 
so  rood,  let  it  do  all  the  work").  Some  instructors  take  a 
while  to  adiust  to  the  idea  that  they  are  no  longer  the 
"main  attraction"  knowing  all  the  information  and  delivering 
it  from  high  atop  the  lecture  platform.  In  fact  the  role  of 
an  instructor  is  much  more  similar  to  that  of  a private 
tutor.  Once  instructors  adopt  this  view  of  their  role,  they 
generally  find  it  rather  rewarding. 


Is  the  course  presently  organized  into  modules, 
chapters,  and  lessons? 


These  mav  provide  convenient  divisions  for  allocating 
assignments,  scheduling  production,  and  organizing  changes 
to  a course. 


Are  course  objectives  available?  Are  old  tests  and 
exams  availabie?  Lesson  plans? 


If  new  staff  members  are  to  develop  course  segments,  such 
docum.ents  and  information  are  extremely  valuable  for 
providing  directions.  Developers  should  be  cautioned,  however, 
that  following  old  lessons  or  lecture  notes  too  closely  can 
be  far  too  limiting.  Old  tests  can  be  used  to  indicate  if 
the  same  level  of  mastery  is  being  achieved  as  well  as 
perhaps  providing  some  provocative  questions  for  the  CBE 
materials.  It  is  worth  some  effort  to  retrieve  such  tests. 


What  media  and  strategies  are  used  in  current  course? 


Film  or  video  cassettes,  filmloops, audio  tapes.  Often 

the  effectiveness  of  existing  materials  can  be  increased  by 
teaming  them  up  with  CBE.  For  example,  a filmloop  may 
contain  points  overlooked  by  students.  A computer-composed 
test  may  focus  attention  and  produce  greater  mastery  with 
little  cost. 

Lab  exercises.  In  many  academic  situations,  students 
complain  because  lab  work  seemingly  has  little  relationship 
to  classroom  topics.  This  problem  frequently  occurs  because 
labs  must  be  planned  weeks  in  advance  so  that  equipment  and 
supplies  can  be  fresh  and  ready.  Variations  in  the  class 
progress  means  that  the  lab  and  lecture  get  out  of  synchron- 
ization. A potential  solution  is  to  tie  a short  CBE  lesson 
to  each  lab  in  order  to  relate  it  to  topics  in  the 
mainstream  of  study. 

Pre-lab  simulations  can  familiarize  students  with  the 
objectives,  apparatus,  and  procedures  associated  with  an 
exercise.  This  solves  the  problem  of  students  who  don't 
read  the  lab  directions  befo  e thf»  lab,  speeds  equipment 
set-up  thus  allowing  longer  exercises  to  be  completed,  and 
reduces  damage  to  equipment  because  of  misunderstanding  and 
haste. 

VJhen  the  purpose  of  a lab  is  to  collect,  analyze,  and 
report  data,  CBF  lessons  can  reduce  the  instructor's  lab 
book  grading  time  by  accepting  data,  checking  calculations 
and  analyses,  and  performing  grading. 

Drill.  A community  college  math  instructor  reported 
that  he  once  spent  a lot  of  class  time  drilling  students. 

He  now  can  use  class  time  to  present  concepts  and  he  lectures 
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only  once  or  twice  a weel;  while  letting  the  computer  super- 
vise the  drills.  Tn  other  cases,  complex  algorithms  for 
determining  when  a student  should  review  missed  items  have 
been  easily  managed  by  the  computer. 


yhat  are  the  outstanding  strengths  and  weaknesses  of 
the  course? 


An  analysis  of  strengths  and  weaknesses  should  point 
out  the  parts  of  the  course  which  should  be  changed,  the 
parts  which  should  be  retained,  and  may  suggest  a direction 
to  start  when  looking  for  solutions  to  course  problems. 

If  a course  is  known  for  its  fine  case  studies,  perhaps 
new  case  studies  which  are  individually  interactive  could 
be  added.  On  the  other  hand,  if  the  course  is  known  for 
having  fine  lecturers,  they  should  not  be  replaced.  Instead, 
consider  developing  a serie.s  of  lessons  to  grade  bomev/ork 
problems,  provide  nrerequisi tes , or  deliver  supplementary/ 
enrichment  materials. 


What  kinds  and  amounts  of  homework  are  assigned? 


CPF-assisted  homework  is  potentially  a way  to  upgrade 
performance,  but  it  will  not  save  money  directly  because 
homework  time  is  cost-free  in  most  environments. 


Will  CPF  lessons  be  used  during  class  time  or  will  they 
be  considered  like  homework--to  be  done  outside  of  class? 
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Depending  on  the  way  CDF  lessons  are  assigned,  students 
n:oy  treat  CPF-taught  topics  differently.  For  example, 
students  who  are  used  to  out-of-class  v;ork  being  practice 
exercises  for  what  is  covered  in  class  will  treat  out-of- 
class  CPE  assignments  similarly:  students  who  do  not  nor- 
mally complete  homework  assignments  are  likely  to  ignore  CPE 
lessons.  However,  if  most  students  normally  perform  out-of- 
class assignments,  new  topics  and  additional  material  may  be 
taught  via  CBF  without  using  class  time  <3F3.**>. 
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Will  the  addition  of  CPF  expand  the  existing  course? 

; 

---  - _ - - - 

CPF  developers  sometime  forget  that  adding  adjunct 
simulations,  practice  exercises,  computer-aided  problem 
solving,  mastery  drills,  etc.,  will  expand  the  amount  of 
student  time  required  to  complete  the  course.  Although  this 

can  sometimes  be  offset  by  providing  labor-saving  routines  j 

for  statistical  analysis,  etc.,  the  overall  impact  on  the 
course  length  should  be  examined. 


What  is  the  student  population  for  this  cour se--homogen- 
eous  or  diverse?  Does  the  course  have  prerequisites? 

Is  there  a need  for  remediation? 


Out-of-class  remediation  for  a handful  of  students  may 
be  a cost-effective  technique  for  reducing  the  completion 
time  of  a group-paced  class  or  for  reducing  the  dropout  rate. 


51 


Is  the  drop-out  rate  large  and/or  prohlenatic? 

The  answer  depends  on  the  situation.  In  a public  edu- 
cation institution,  the  dropout  rate  is  probably  irrelevant 
unless  it  is  embarrassingly  large  or  unless  departmental 
funding  is  dependent  on  the  number  of  students  completing 
the  course.  In  military  and  industrial  training  institutions 
and  in  privately-supported  colleges,  dropouts  are  typically 
expensive.  CBf  materials  may  be  part  of  a potential  cure 
for  a dropout  problem. 


Is  the  course  taught  to  a level  of  mastery  and/or  are 
grades  assigned? 


CBF  is  ideally  suited  for  teaching  to  a mastery  level, 
but  some  institutions  require  some  sort  of  score,  grade  or 
other  discrimination  device. 

The  decision  to  teach  to  a mastery  level  requires  more 
different  forms  of  tests  than  are  typically  used  in  a 
non-mastery  situation  since  re-testing  to  show  mastery  is 
necessary . 

It  should  be  noted  that  some  students  prefer  HOT  to 
achieve  mastery  of  the  material  and  find  it  frustrating 
when  they  cannot  relax  and  take  a "gentleman’s  C"  for  the 
course.  Whatever  the  medium  used,  the  introduction  of 
mastery  requirements  eliminates  traditional  grading.  Fssen- 
tially,  everybody  gets  the  same  grade! 
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Will  CPE  lessons  be  phased  in  as  they  are  completed, 
or  will  the  course  be  suddenly  converted? 


Tidy  research  designs  and  the  practicalities  of  forma- 
tive evaluation  and  ooerational  teaching  prescribe  different 
answers.  For  "tidy"  research,  the  full  complement  of  CRF 
lessons  should  be  implemented  at  once  (to  all  students  or  to 
half  of  a randomly-selected  group  of  students).  From  the 
point  of  view  of  a department  or  instructor  trying  to  upgrade 
the  training  or  solve  teaching  problems  or  a curriculum 
developer  wishing  to  tune  the  courseware  to  the  students, 
the  favored  approach  is  more  likely  to  be  to  make  a subjec- 
tive evaluation  of  the  value  of  new  materials  and  implement 
immediately  all  those  which  seem  completely  debugged  <2D1.5>. 


53 


Who  will  decide  what  goes  on  to  the  CDE  system?  What 
criteria  will  be  used? 


As  indicated  previously,  this  question  is  of  critical 
importance.  If  instructors  or  institutions  distant  from  the 
developers  are  expected  to  use  the  materials,  this  is  a 
logical  point  for  requesting  their  input.  The  criteria  for 
choosing  topics  for  CHE  development  depend  or,  the  project 
objectives,  modified  by  the  considerations  of  what  CEE  can 
do  especially  well.  As  much  experience  as  possible  should 
be  employed  when  making  these  decisions.  This  is  one  of  the 
times  during  a project  when  an  outside  consultant  may  be 
needed . 


Will  the  lessons  written  be  considered  to  "belong"  to 
a course  or  to  an  instructor-developer?  Will  future 
instructors  be  required  to  use  some,  any,  or  all  CBE 
lessons  written  by  this  organization? 


In  a higher  education  environment,  if  the  lessons  are 
to  be  written  for  a specific  course  to  be  used  by  current  or 
subsequent  instructors,  it  may  be  important  that  potential 
users  be  involved  at  all  stages.  Requesting  reviews  from 
non-CBF  instructors  is  often  a good  first  step.  Failure  to 
involve  others  may  result  in  unused  lessons  when  other 
instructors  teach  the  course.  In  a military  or  pt imary- 
secondary  public  education  facility,  instructors  have  less 
choice  about  teaching  materials  and  the  problem  outlined 
here  occurs  less  frequently  <3F1.1,  3F1.2,  3F1.3,  3F1.4, 
3F2.2,  3F2.3>. 
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Will  lessons  be  written  so  they  can  be  used  in  several 
different  courses? 


On  the  surface,  this  path  seems  prudent  because  it 
allows  development  costs  to  be  amortized  more  quickly.  But 
there  are  some  hidden  pitfalls.  Normally  the  backgrounds  of 
students  in  the  different  courses  will  vary  considerably. 

ne  group  of  students  will  be  bored  or  the  other  strained 
unless  Herculean  efforts  at  individualization  and  ability 
tracking  are  made.  These  efforts,  of  course,  bi  -^st  develop- 
ment time. 

A better  approach  is  to  write  lessons  for  higher  ability 
students,  test  and  polish  it  for  them.  Next,  have  a few 
students  from  another  course  use  the  lesson.  Note  their 
problems,  and  modify  a copy  of  the  lesson  to  meet  the  differ- 
ing prerequisites  of  the  other  users.  In  many  cases  less- 
prepared  students  will  point  out  difficulties  which  should 
be  repaired  in  all  versions  of  the  lessons.  Nevertheless, 
it  is  so  much  simpler  to  write  a lesson  for  a single,  narrow 
audience  that  it  is  worth  the  slight  additional  computer 
storage  and  modification  time  needed  for  multiple  versions. 
Furthermore,  the  author  will  not  waste  any  time  installing 
extra  help  and  remediation  which  are  not  truly  needed. 
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THE  STUDENT  POPULATION 

How  are  students  selected  for  the  course?  Are  any 
predictive  tests  administered? 

Knowledge  about  the  students  is  important  since  many 
design  decisions  are  based  on  the  breadth  and  depth  of  student 
abilities.  Listed  prerequisites  are  often  out  of  date  and 
should  not  be  accepted  without  examination  of  current  class 
makeup  and/or  future  needs.  In  technical  and  vocational 
courses,  reading  level  and  math  skills  should  be  checked  if 
possible.  On  the  other  hand,  don't  spend  time  administering 
or  retrieving,  test  data  (e.g.,  SAT  scores)  unless  the  use 
for  them  is  clearly  established. 

Is  motivation  a problem  for  students? 

If  so,  improved  motivation  may  be  an  objective  to  be 
achieved  via  CBE.  However,  do  not  expect  improvement 
merely  because  of  the  change  of  medium.  Few  improvements 
are  automatic.  One  must  use  strategies  to  enhance  the 
relevancy  or  increase  student  involvement,  for  example. 

What  problems  do  students  have  during  and  after  this 
course? 

The  effect  of  CPE  can  be  more  than  proportionate  to  the 
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fraction  of  time  spent  usinp  it  if  bottlenecks  can  be  allev- 
iated and  hurdles  can  be  lowered.  This  is  also  a fine  wav 
to  enhance  motivation. 


Ts  terminal  use  to  be  scheduled  or  free  flowing? 


Since  CRF  is  inherently  self-paced,  it  is  convenient 
if  students  can  use  terminals  freely.  Thus  they  can  skip 
classes  when  they  are  ahead  and  they  can  find  vacant  terminals 
when  they  are  behind.  In  any  case,  some  convenient  unsched- 
uled time  should  be  provided  for  slower  students.  This 
brings  up  an  interesting  semantic  problem.  The  term  "slow 
student"  has  become  a euphemism  for  "poor  student."  Another 
term  must  be  found  to  describe  students  in  a self-paced 
mastery  situation  where  "slowness"  carries  a much  smaller 
negative  connotation. 


Do  students  need  additional  rewards  for  achieving  high 
per f ormance? 


For  example,  some  sites  allow  students  to  play  games  as 
a reward.  It  is  probably  as  motivating  (and  certainly 
cheaper)  to  install  a publicly-available  "record  book"  for 
displaying  the  score  of  high  achievers  within  instructional 
lessons.  One  successful  format  incorporates  programming 
to  eliminate  the  recording  of  embarrassingly  low  scores  and 
to  cause  the  premature  obsolence  of,  say,  the  sixth  through 
tenth  best  scores.  The  latter  feature  deletes  some  old 
scores  so  that  achieving  a "recordable"  score  does  not 
become  impossible. 
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l!ow  pressured  are  the  students  for  time?  for  grades? 


CPF  will  not  be  popular  if  students  already  pressed 
for  time  perceive  that  CDE  assignments  take  longer  than 
assignments  for  conventional  media.  Similarly,  the  use  of 
CRE- given  grades  (or  grades  from  any  mastery-type  training) 
must  be  carefully  considered  for  students  who  are  especially 
anxious  about  grades  (e.g.,  pre-med  students).  It  is  gen- 
erally inappropriate  to  "curve"  scores  from  mastery  exercises 
in  order  to  provide  "grades."  Another  problem  with  grades 
based  on  performance  in  a CRF  lesson  is  that  some  lessons 
can  be  repeated  by  a student.  Which  grade  should  be  recorded; 
The  first?  The  last?  The  average  of  all  attempts? 
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DEVELOPING  A CURRICULUM 


Are  you  preparing  a curriculum,  not  just  lessons? 


Although  some  reasonable  uses  of  CRE  involve  the 
creation  of  a single  lesson  not  terribly  closely  related  to 
other  topics  being  taught,  one  usually  wishes  to  integrate 
the  CBE  lessons  with  each  other  and  with  materials  in  other 
media.  However,  because  many  users  start  out  thinking  small 
(especially  at  a general-support  site),  it  frequently  happens 
that  a course  eventually  contains  a collection  of  lessons 
which  cannot  be  transformed  into  a curriculum. 

It  is  folly  to  develop  a lesson  without  adequately 
planning  for  its  integration  into  an  educational  context. 

If  you  are  not  planning  to  modify  a whole  course,  but  rather 
are  creating  a few  lessons  to  present  some  d if f icul t-to- 
teach  topics,  the  concept  of  a mini-course  may  he  the  best 
approach.  A mini-course  is  an  integrated  cluster  of  on-line 
and  off-line  materials  which  teaches  a single,  limited -range 
topic.  The  critical  features  of  such  a cluster  are:  (a) 
well-defined,  but  limited  prerequisites  (b)  consistent 
terminology,  objectives,  and  perspective  and  (c)  a definite 
ending  point  (from  which  another  mini-course  can  he  launched). 

Creating  lessons  without  a master  plan  can  be  ineffi- 
cient. For  example,  one  developer  explored  the  use  of  the 
CRE  system  writing  several  kinds  and  styles  of  lessons.  He 
introduced  the  CRE  system,  the  keyboard,  and  checked  the 
student's  math  skills  in  each.  As  he  developed  clusters  of 
lessons,  he  found  that  the  students  were  being  re-introduced 
to  the  CBE  system  (and  having  their  math  skills  tested) 
again  and  again.  ^furthermore , several  lessons  just  would 
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not  "fit"  into  a duster.  'Eventually  all  lessons  had  their 
introductions  removed,  and  some  lessons  were  divided  into 
several  parts  so  they  could  fit  into  clusters. 


Will  the  lessons  be  used  for  mainline  instruction  or 
as  an  adjunct  to  other  instruction? 


Mainline  lessons  provide  an  easier  format  to  justify 
costs  and  show  savings,  but  they  require  more  care  and  test- 
ing, during  creation  and  implementation  (because  they  are 
the  sole  source  of  information  about  a topic).  If  sufficient 
time  is  available,  one  pata  which  is  often  successful  is  to 
use  the  lessons  in  an  adjunct  mode  initially  until  any  prob- 
lems are  eliminated  (or  a skeotical  audience  is  won  over)  and 
then  usf'  them  as  mainline  lessons  replacing  conventional 
materials.  This  conver'^ion  from  "adjunct"  to  "mainline" 
shou''d  be  phased  in  beginning  as  soon  as  possible  because 
student  standards  for  each  type  of  usage  are  likely  to  be 
different  and  the  elimination  of  the  previous  mainline 
materials  may  reveal  some  oversights. 


Are  you  modifying  an  existing  curriculum,  or  creating 
a new  one? 


‘'ost  people  realize  that  unless  they  modify  their 
curriculum  somewhat,  they  won't  be  able  to  make  good  use  of 
their  CPF  system;  fewer  realize  that  if  their  mod  if i cations 
are  too  ambitious,  they  may  not  be  able  to  handle  the  twin 
tasks  of  curriculum  development  and  mastering  a now  medium. 
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The  decision  about  where  to  stribe  r.  balanc<^  deoends  on  the 
experience  of  the  proup  with  respeci  to  curriculum  develop- 
ment and  fhF  instruct iona]  design  as  well  as  the  manpower 

available. 

because  of  the  vast  possibilities  and  the  great  number 
of  choices  available  from  a CfF-based  curriculum,  sites  with 
moderate  curriculum  design  experience  •'in  limited  media)  have 
become  mired  and  have  fallen  short  of  their  goals.  Further- 
more, some  kinds  of  evaluation  are  virtually  impossible  if  a 
major  revision  is  made  simultaneously  with  a change’  in  media. 
Although  writing  an  entirely  new  course  is  a very  diffi- 
jl  cu^t  task,  t!ie  result  may  not  be  so  threateninr  to  current 

I instructors.  ’furthermore,  one  may  be  able  to  use  tbe  f'F 

I system  to  avoid  capital  costs  associated  with  labs  or 

1 demonstration  devices. 


Vi  11  media  other  than  C’^’f  be  used?  Who  vn  1 1 be 
responsible  for  deciding  v;hat  media  to  use  for  each 
topic  and  tor  insuring  that  the  media  are  interfaced? 


^hcre  are  signiticant  problems  with  teaching,  ’arge 
nortions  of  a course  only  via  mr  lessons:  rPF  resources  are 
expensive,  thev  are  inappropriate  for  some  uses,  and  tbev 
have  unnee'’ed  power  for  .some  applications.  ’“ixing  media  i .s 
more  efficient,  but  requires  someone  to  integrate  from  the 
beginning  all  materials  written  on  a topic 


What  offline  materials  or  paper  copies  of  CHF  lessons 
will  bo  made  available?  on  what  conditions? 
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One  of  the  greatest  shortcomings  of  PLi^TO  CPF;  materials 
produced  so  far  is  that  they  attempt,  unsuccessfully,  to  he 
totally  stand-alone.  Although  providing  manuals,  worksheets, 
study  guides,  outlines,  and  summaries  to  accomoany 
lessons  is  very  sensible,  it  is  seldom  done.  ''s  a result 
the  cost-e tfecti veness  of  some  applications  may  be  reduced. 
For  example,  students  who  want  to  have  something  to  study 
after  thev  have  left  the  terminal  are  forced  to  take  notes, 
hote-taking  is  an  effective  study  strategy  for  some  kinds 
of  learning,  but  cheaper  alternatives  than  copving  from 
a CPF  terminal  are  available.  Forcing  students  to  take 
notes  from  the  terminal  may  change  a lesson  which  saves 
tine  (and  money)  into  a lesson  which  takes  extra  time. 

Instructors  may  fear  that  students  will  skip  the  fPF 
lesson  if  they  can  pick  un  a summary  of  what  they  must  learn. 
If  a summary  sheet  can  adequately  express  all  the  concepts 
which  must  he  learned,  there  is  no  reason  that  the  CPF 
lesson  should  he  used.  If  the  CP*^  lesson  is  truly  necessary, 
but  the  students  are  lazy,  the  instructor  might  wish  to  make 
summaries  available  only  after  the  student  has  successfully 
completed  the  lesson  or  only  several  days  before  major  tests 
<kF1 . P>  . 


'•'ill  published  lessons  (written  at  other  sites) 
comprise  a significant  amount  of  the  courseware  used'!’ 


Y_e_s.  It  is  a major  task  to  interface  the  topics  and 
courses  taught  at  one's  own  site  with  lessons  selected  from 
curricula  written  elsewhere.  "omeonc  should  be  given  responsi- 
bility for  gathering  lists  of  available  materials  and  aiding 
instructors  in  the  selection  of  lessons  based  on  more  than 
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their  titles.  As  in  textbook  selection,  a problem  sometimes 
occurs  because  materials  which  are  optimized  for  students  may 
seem  unst  imul ati n®  and  unappealine  to  an  instructor  <?n2.5>. 
Suggestion:  include  several  students  on  the  lesson  selection 

committee . 

Modification  of  lessons  written  at  other  sites  often 
entails  a great  deal  of  work  unless  the  changes  involve  only 
the  addition  or  deletion  of  displays  and  text.  Documentation 
is  frequently  lacking  and  there  may  be  problems  in  finding 
the  author  and  receiving  permission  to  modify  his  work. 

Mo . Unless  terminal  delivery  is  phased,  the  decision 
not  to  use  materials  written  elsewhere  will  result  in  a 
long  period  of  under-utilized  terminals.  Furthermore,  the 
relatively  large  development  costs  associated  with  CDp 
mav  threaten  cost-effectiveness  if  some  existing  lessons 
cannot  be  employed.  An  insightful  comment  by  a well-known 
educator  unfortunately  applies  to  many  CPE  develoners: 
"Researchers  and  educators  frequently  demonstrate  a strong 
resistance  to  the  use  of  someone  else's  innovation.  Tt  has 
been  said  that  if  there  were  a Mobel  prize  for  educational 
research,  we  would  have  to  nominate  an  entire  generation 
of  researchers  for  their  co-discovery  of  the  wheel," 

( Molnar  , 1971). 


Are  lessons  to  be  written  simultaneously  for  all  parts 
of  the  course  or  will  you  start  from  one  end  working 
toward  the  other'’  Which  end  wi’l  you  start  from? 


All  parts  simultaneously.  This  may  be  necessary  if 
several  authors  with  different  sub-specialties  are  employed. 
.Spreading  the  effort  may  make  testing  more  difficult  and 
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m?y  require  more  effort  to  assure  that  objectives  and 
prereni’i  si  tes  match  up. 

^ec’i_nnin_g  ?Ill'  '•■'ben  lari’.e,  consecutive  portions 

of  a course  are  to  be  rewritten  to  include  CPE  instruction, 
it  is  frequently  advantageous  to  begin  lesson  v/ritinp  and 
implementation  at  the  end  of  the  course.  This  intuitively- 
wronc  order  Has  the  following  advantages: 

1.  "^he  first  material.s  seen  by  a student  (those  most 
likely  to  turn  him  on  or  off  to  CBF)  are  those  written 
when  the  authors  and  staff  are  most  experienced  in  CPP 
deyelopment . 


?.  The  nrocess  follows  the  order  of  a formal  task 
analvsis'  if  mid-way  through  the  development,  one 
finds  students  unable  to  understand  the  material,  one 
can  easily  add  prerequisite  topics  because  the 
development  of  prerequisite  topics  is  alvjays  "down  the 
road"  in  terms  of  tine.  If  a part  of  the  task  analysis 
must  be  postponed,  development  can  nevertheless 
start  immediately. 

?.  Tf  the  course  is  being  converted  from  lock-step 
to  self-paced  as  the  CBF.  materials  are  introduced  (a 
common  occurrence),  students  can  move  through  the 
conventional  materials  together,  then  finish  the 
self-paced  part  of  the  course  at  their  own  rate.  If 
the  course  were  rewritten  beginning  at  the  start, 
students  using  the  self-paced  CPF  materials  would  have 
to  catch  up  or  wait  for  the  whole  class  to  Hegin  the 
conventional  lock-step  section  of  the  course.  Any 


time  savings  realized  from  self-pacing  and  CBE  lessons 
can  be  taken  advantage  of,  even  during  preliminary 
testing  phases. 

II.  If  for  some  reason  the  development  effort  falls  short 
of  its  goals  in  terms  of  quantity  of  lessons  produced, 
it  is  less  likely  that  the  evaluation  plan  will  be 
completely  invalidated  or  impaired.  As  noted  above, 
actual  time  savings  can  be  measured,  not  estimated; 
CPF-influenced  attitudes  will  be  "fresh":  and  the 
higher-level  cognitive  skills  (of ten  taught  near  the 
end  of  the  course)  for  which  CRF  can  often  show  cost- 
effectiveness  will  be  present  in  the  completed  portion 
of  the  course . 
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I.FSSON  TYP'^F 

f 

'•'ill  tests,  as  well  as  lessons,  be  prof^ramiried  for  CRE 
u s a p,  e ? 


Fonie  sites  have  used  CPF  almost  exclusively  for 
testing  ^'ecause  of  the  capability  for  computer-generated 
test  items,  easy  recording  of  test  item  statistics  (e.g., 
reliability,  difficulty  level,  discrimination  index),  and 
the  opportunity  for  feedback  ('luerra,  McRaniel,  Kaufman  ^ 
Fuller,  1977).  Other  instructors  have  felt  more  comfortable 
using  lessons,  but  keeping  their  tests  in  a conventional 

format.  Ftill  other  users,  faced  with  an  evaluation  plan 
calling  for  a comparison  of  students  taught  via  CPF  lessons 
with  students  taught  conventionally,  have  realized  they  must 
test  conventionally  in  order  not  to  give  unfair  advantage  to 
students  Familiar  with  the  CRF  system. 

Vhat  problems  with  test  security  exist'’ 

To  avoid  embarrassment,  test  security  should  be  checked 
thoroughly  before  items  are  entered  on-line.  Procedures  for 
checking  student  identification  should  be  planned  and  built 
into  tests.  /'llowance  for  system  failures  during  testing 
must  be  made. 
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LFSSCM  r.TA^'DAPnr^ 


'■'ill  all  lessons  producerl  on-site  have  to  meet  a set 
of  standards? 


Tn  (general  the  idea  of  "maintaining  standards"  is  a 
sound,  practical  one.  Tt  may  ba  difficult,  however,  to 
impose  such  standards  on  professionals  using  the  terminals 
at  a genera] -support  site.  Fortunately,  the  need  for  strong, 
standards  is  greatest  in  a direct-support  project  situation. 
The  question  of  creativity  vs.  control  cannot  yet  be  answered 
easily  and  objectively.  An  affirmative  answer  to  the  above 
Query  naturally  raises  tt^e  issue  of  what  standards  will  be 
set,  who  will  enforce  them,  and  how  the  enforcement  will  be 
implemented.  Previous  attempts  to  impose  standards  by  regu- 
lating storage  space  for  new  lessons  have  quickly  raised 
authors'  ire.  Tn  other  cases,  the  "enforcer"  had  a lower  pay 
or  rank  than  the  authors  and  programmer  and  hence  was  it^nored 
by  them. 

It  is  over ly-optimistic  to  expect  new  staff  to  prepare 
polished  lessons  in  their  first  attempt.  Phased-in  standards 
are  likely  to  be  more  appropriate.  For  example,  the  t rst 
lesson  might  be  written  without  the  imposition  of  any  formal 
standards.  When  the  second  lesson  is  written,  a copy  of 
the  standards  could  he  provided,  along,  with  a justification 
for  using  them.  As  later  lessons  are  prepared,  the  use  of 
standards  would  be  encouraged  and  enforced.  Tf  necessary, 
the  initial  lessons  written  would  be  brought  within  the 
guidelines.  hrawing  up  the  standards  with  the  aid  of  the 
author  staff  promotes  motivation  and  understanding  of  the 
need  and  use  of  lesson  standards. 


Vhat  l^inds  of  standards  mip,ht  he  imposed'’  What  purpose 
does  each  serve? 


documentation . anoropriate  level  of  documentation  f 

protects  the  investment  of  the  sponsor,  agency,  or  department  , 

by  insuring  that  another  user  wishing  to  update,  correct, 
or  modify  the  programming  can  do  so  in  an  efficient,  reliable 
manner.  It  also  insures  that  the  author  himself  will  not 
waste  time  trying  to  interpret  programs  he  wrote  long  ago. 

standards  for  IiCS  , CPU,  and  HAhM.  These  are  abbrevia- 
tions for  lesson  size  and  the  demands  which  an  operating  les- 
son places  on  the  computer.  tf  the  demands  are  excessive, 
the  lessons  will  he  unusable,  ineffective,  or  will  reduce  the 
lesson  access  for  others  at  a site.  .Standards  for  usage  of 
computer  resources  such  as  these  insure  that  the  reputation 
of  the  site,  authors,  and  system  will  be  protected  and  that 
students  trying  to  learn  from  the  lessons  will  not  be  annoyed 
or  impeded. 

Lesson  format.  Several  aspects  of  a lesson's  format 
might  be  subject  to  standards.  If  strategies  inappropriate 
to  CBS  are  used,  cost-effectiveness  will  become  a problem 
and  expensive  development  time  will  be  wasted.  For  example, 
lecture-like  lessons  are  sure  "losers"  yet  are  commonly 
prepared  by  some  authors.  The  presence  of  a title  page, 
statement  of  objectives,  and  author  name  on  each  lesson  can 
provide  a cohesiveness  among  lessons  which  students  find 
comfortable  and  efficient.  The  inclusion  of  restart  points 
protects  a student  from  system  failures  and  allovjs  him  to 
leave  the  system  and  return  without  re-doing  part  of  the 
lesson.  By  including  data  collection  standards,  a general 
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support  site  can  insure  that  authors  who  use  site  resources 
also  generate  information  to  nrovide  a basis  or  justification 
for  the  continued  support  of  the  site.  Data  collection 
standards  can  also  prevent  errors  in  lessons  from  going 
undetected  and  plaguing  subsequent  groups  of  students. 

Some  sites  include  a requirement  for  a criterion- 
referenced  test  at  the  end  of  each  lesson.  Properly  construc- 
ted, such  a test  will  allow  a student  to  get  "credit"  for  a 
lesson  only  after  mastery  of  the  material  has  been  demon- 
strated. Without  such  a test,  some  students  feel  that 
completion  of  the  lesson  means  that  further  study  of  the 
topic  is  unnecessary.  Authors  of  succeeding  lessons  may 
find  that  students  are  unable  to  understand  those  lessons 
because  of  inadequate  mastery  of  the  prerequisites. 

Most  sites  include  some  sort  of  subject  matter  and 
editorial  standards  as  well  in  order  to  assure  that  all 
facts  are  accurately  stated  and  all  grammatical,  semantic, 
spelling,  and  typographical  errors  removed.  rj 


If  lessons  from  other  sites  are  used,  will  they  have 
to  meet  the  same  criteria?  Who  will  modify  and 
unerade  them? 
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MAMAGinC.  FORMATIVF  DEVELOPMENT  (EVALUATING  LESSONS) 

What  criteria  will  be  used  to  determine  when  a lesson 
is  finished  or  validated? 


Although  few  lessons  are  ever  truly  "finished,"  we  use 
that  word  here  to  indicate  that  a lesson  may  be  used  by  a 
class  of  casual ly-proctored  students.  Some  combination  of 
student  use  (involving  both  the  total  number  of  students 
and  the  number  of  testing/revision  cycles)  and  colleague 
review  should  be  established  (and  adjusted  with  experience) 
so  that  lessons  do  not  reach  students  prematurely. 


How  will  lessons  be  evaluated?  I'ow  often  will  they  be 
evaluated?  Will  they  be  examined  after  they  are 
finished  or  validated? 


Evaluations  during  both  formative  development  and  as 
part  of  a summative  development  are  warranted.  At  some 
sites  nationally-known  experts  in  a field  are  brought  in  to 
review  lessons  which  the  site  staff  consider  finished. 

The  aspects  of  a lesson  which  are  normally  evaluated 
include:  the  programming,  the  instructional  design,  the 


conformance  to  site  standards,  and  the  subject  matter  accuracy. 
In  many  cases  the  evaluation  is  broadened  to  include  student 


performance  results,  student  and  instructor  comments,  and 
data  concerning  the  quantity  of  usage.  For  a more  detailed 
discussion  of  the  aspects  of  formative  development  reviews, 
see  Francis,  Goldstein,  and  Call-Himwich  (1975)  and  Call- 
Himwich  (1976). 


70 


I 

I 

I 

■ j 


Are  student  "guinea  nigs"  for  testing  lessons  readily 
available?  V'hose  responsibility  is  it  to  provide  them? 
What  rewards  can  they  be  offered  for  their  cooperation? 


A group  of  eager  students,  possessing  about  the  right 
background  for  the  material  and  a readiness  to  try  out  lesson 
^r_agmen^ts,  can  be  an  enormously  valuable  asset  for  a site. 

The  absence  of  such  a ponulation  allows  authors  to  create  too 
much  material  before  testing  it  with  students.  Since  authors 
tend  to  do  this  under  any  circumstances,  one  must  present 
them  wit-h  many  easy  opportunities  to  obtain  student  feedback. 

If  sufficient  terminal  time  is  available,  it  may  be 
possible  to  "pay"  students  bv  giving  them  time  in  recreational 
lessons.  The  amount  of  time  they  earn  is  proportional  to  the 
number  and  value  of  their  comments. 


Are  lesson  objectives  available?  Will  they  be  written? 
Who  will  write  them  and  when  will  they  be  written? 


I 

There  are  advantages  to  choosing  a course  for  which  the 
objectives  have  already  been  prepared.  It  is  easy  to  be 
deceived,  however,  unless  one  obtains  copies  of  the  objectives 
and  reviews  them  carefully:  the  written  objectives  are  often 
outdated  and  do  not  reflect  the  content  and  emphasis  of  the 
course.  In  many  cases  they  are  so  general  that  they  have 
little  value,  written  objectives  also  tend  to  record  only 
those  topics  which  can  be  explicitly  taught  and  tested. 

Looking  at  such  too-narrow  objectives  may  limit  the  potential 
applications  of  CBF.  For  example,  the  goal  of  having  a 
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student  ask  questions,  plan  an  invest i Ration , and  carry  it 
out  may  be  an  unstated  and  unmet  goal  which  CBE  exercises 
could  provide.  Similarly,  the  objective  of  training  for 
discrimination  of  sounds  may  have  been  deleted  because 
appropriate  hardware  was  not  available  prior  to  the  installa- 
tion of  the  CBE  system. 


Vho  is  responsible  for  verifying  that  the  objectives  are 
appropriate  in  terms  of  content  and  level  of  knowledge? 


Are  objectives  needed? 


This  question  arises  frequently,  especially  among  staff 
who  have  had  unpleasant  experiences  with  writing  and  using 
objectives.  Former  classroom  teachers  may  have  been  forced 
to  use  objectives  by  administrative  fiat  and  may  harbor  a 
grudge  against  objectives  because  they  seem  useless. 

Furthermore,  it  is  true  that  learning  objectives  are 
sometimes  so  obvious  they  need  not  be  stated.  For  example, 
the  need  for  students  to  solve  a particular  type  of  problem 
may  b**  well  recognized  (e.g.,  two-n  imensional , frictionless, 
collision  problems  or  non-disproportionation,  oxidation- 
reduction  equation  balancing).  Tn  these  cases  the  objective 
is  to  be  able  to  solve  any  problem  of  that  type  and  the  test 
is  to  give  the  student  a problem  and  let  him  solve  it  without 
aid.  A more  formal,  written  objective  is  not  needed  because 
any  subject  matter  expert  seeing  the  lesson  topic  could 
immediately  determine  the  material  being  taught. 
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Typically,  the  lesson  developer  is  not  so  lucky  as  to 
have  such  "tidy"  topics  to  teach.  Instead,  the  depth  of  the 
instruction  (Are  exceptions  to  the  rule  to  be  included?)  and 
the  breadth  of  instruction  (Where  does  this  lesson  end?)  must  be 
determined  as  part  of  the  lesson  development  process.  'because 
a CBF  lesson  will  probably  interact  with  more  students  than  a 
classroom  instructor,  the  CPF  developer  can  and  should  spend 
more  time  in  the  planning  and  objective  writing  than  the 
lecturer.  A well-written  set  of  lesson  objectives  can 
provide  the  developer  with  guidance  and  act  as  a touchstone 
so  the  author  knows  what  is  relevant  to  the  lesson  and  what 
is  superfluous,  when  the  lesson  is  done,  and  how  to  determine 
if  the  lesson  is  adequate. 


I 

i 


I 
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i 
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! PART  1 1 -FUNDAMENTALS  OF  CHOOSING  AND  TRAINING  A CBE  STAFF 
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FSSEMTIAL  SKILLS 

When  considering  what  staff  to  hire,  an  administrator 
typically  begins  by  forming  job  descriptions,  then  looking 
for  people  who  match  the  job  descriptions.  Because  of  the 
limited  availability  of  staff  with  CBE-related  experience, 
we  suggest  considering  the  skills  needed  for  the  project 
rather  than  roles  or  job  titles  to  be  filled.  Below  we  have 
listed  ten  skills  typically  needed  within  a CBF  development 
staff.  In  a small  project,  two  or  three  people  may  have  to 
provide  all  the  skills  listed;  even  in  a large  project, 
several  roles  are  generally  combined.  Because  skills  are 
present  to  varying  degrees  in  different  staff  members, 
premature  combining  of  skills  into  a job  description  should 
be  avoided.  Analyze  the  .'ontribution  of  each  candidate  in 
terms  of  which  project  needs  he  could  serve  and  what 
flexibility  his  talents  would  offer. 

Administrative/ managerial.  This  skill  is  needed  to 
coordinate  the  project,  maintain  standards,  and  expedite 
non-clerical  details.  This  is  a general  project  support 
role--to  smooth  the  path  and  monitor  progress.  It  entails 
less  decision-making  than  the  skill  of  leadership,  discussed 
next . 

Leadership . This  important  skill  provides  direction, 
gives  rewards  or  prods,  sets  a pace  or  an  example,  makes 
decisions,  and  initiates  changes. 

Curriculum  coordination.  This  skill  is  necessary  to 
insure  that  the  curricula  or  modules  produced  are  well- 
integrated  packages.  It  requires  the  broadest  subject  matter 
experience.  Although  this  skill  requires  much  coordinating, 
it  is  not  necessarily  the  manager's  function. 

expertise.  This  skill  is  employed  when 

preparing  objectives,  providing  content,  writing  text,  choos- 
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inp  examples  and  non-examples,  describing  typical  misconcep- 
tions, pointing  out  exceptions  to  rules,  etc. 

Tns_triijct^  exp^r ienc^e . This  skill  is  necessary  for 
providing  insight  about  the  students'  backgrounds,  problems, 
motivations,  futures,  and  typical  complaints. 

2l2st  r uc^qna  l_^^gn  . This  often  overlooked  or  under- 
estimated skill  provides  knowledge  about  the  process  of 
planning,  designing,  implementing,  and  revising  instructional 
materials . 

Evaluation /testing.  These  two  skills  each  probably 
deserve  categories  of  their  own,  but  their  relatedness  means 
that  they  are  often  possessed  by  a single  person.  These 
skills  serve  to  promote,  implement,  and  aid  formative  and 
summative  evaluation  practices.  Depending  on  the  project 
objectives,  the  preparation  of  a research  design  and  an 
analysis  of  results  may  also  be  needed. 

£PE/l_angua^  expertise . This  skill  is  needed  in  order 
to  program  lessons  and  advise  in  the  efficient  uses  of  the 
computer.  It  is  also  needed  to  explain  and  exploit  the 
various  communications  features,  the  courseware  design 
considerations,  and  the  system  characteristics  associated 
with  the  CPE  system.  Experience  on  other  computer  systems, 
particularly  other  CBE  systems,  is  a frequently-needed 
compromise  in  staff  selection  which  may  broaden  the  total 
base  of  experience  enjoyed  by  the  site. 

Editorial  skills.  These  skills  enable  a site  to  "polish" 
the  final  product,  to  "debug"  the  non-programming  problems, 
to  sharpen  definitions  and  explanations,  to  correct  grammar 
and  spelling,  and  to  insure  consistency  and  smoothness 
through  the  instructional  materials. 

General  support.  The  following  skills  are  needed  to 
support  the  efforts  of  the  primary  CPE  development  effort: 
the  skills  of  on-site  hardware  repairmen,  proctors,  secretar- 
ies, clerks,  photographers,  and  illustrators. 
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A very  careful  assessment  of  the  skill  needs  and  avail- 
able talent  can  take  advantage  of  the  special  backgrounds  of 
the  staff.  Thus  rather  than  considering  that  one  needs  a 
staff  of  a manager,  secretary,  five  authors,  two  programmers 
and  an  evaluator,  and  that  one  should  find  people  who  best 
meet  those  descriptions,  a wise  administrator  can  choose 
people  whose  experiences  are  complementary  even  if  the 
candidates’  qualifications  do  not  exactly  match  typical  job 
descriptions.  A flexible  policy  in  finding  personnel  might 
allow  an  administrator  to  choose  a staff  composed  of;  a 
subject  matter  expert  (SME)  with  curriculum  development 
experience  in  another  field;  a programmer  with  some  testing 
experience;  a current  course  instructor  who  has  subject 
matter  experience;  an  evaluator  who  has  worked  in  instruc- 
tional design;  a subject  matter  expert  with  computer  science 
courses,  but  no  teaching  experience:  a CBE  author  with  a 
hobby  related  to  the  subject  matter;  a secretary  who  was  a 
Journalism  major  for  a year;  a manager  who  is  an  experienced 
course  instructor  and  has  written  or  revised  many  of  the 
course  materials;  an  SME  with  some  courses  in  learning  and 
instruction  as  well  as  an  artistic  flair;  and  an  SME  who  was 
a technical  v;riter  in  industry. 

We  emphasize  flexibility  in  staff  selection  because  the 
most  consistent  finding  of  a study  of  PLATO  courseware  devel- 
opment procedures  in  military  and  civilian  environments 
(Mahler,  et  al.,  1976)  was  that  courseware  groups  changed 
their  organization  one  or  more  times  during  their  compara- 
tively short  (two  to  three  year)  project  lifetimes.  The 
changes  in  staff  roles  caused  by  reorganization  can  be  some- 
what minimized  by  emphasizing  skills  over  positions.  Skills 
will  change  rather  gradually,  and  provide  continuity  during 
periods  of  reorganization. 
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Staff  reorganizations  caused  by  weaknesses  of  the 
members  are  sometimes  simplified  if  skills  rather  than  job 
titles  are  emphasized.  If  a staff  member  is  unable  to 
perform  satisfactorily  and  discharge  and  replacement  is  not 
feasible,  reassignment  of  duties  may  be  the  answer.  For 
example,  an  author  having  trouble  mastering  the  CBE  language 
might  be  assigned  only  SME  responsibility.  Alternatively, 
that  author  might  be  used  as  a proctor  or  lesson  editor. 


SKILL  COMBINATIONS 


VJhen  choosing  project  staff  from  available  candidate 
pools,  one  is  forced  to  look  for  individuals  who  can  provide 
several  of  the  ten  talents  and  skills  listed  above.  One  must 
also  be  prepared  to  make  compromises  in  filling  positions  and 
consider  the  flexibility  of  individuals  should  their  role  or 
the  project  change.  Suggested  below  are  several  common  skill 
combinations  which  result  in  high  productivity. 

The  leader  of  a project  often  has  the  additional  responsi 
bilities  of  curriculum  coordination,  project  management,  or 
both.  It  is,  however,  important  to  see  these  as  different, 
but  related,  roles.  An  overworked  project  leader  can,  if 
necessary,  assign  these  responsibilities  to  others  while 
still  maintaining  his  authority  and  position. 

Assumption  of  the  leadership  and  evaluation  roles  by 
a single  person  is  most  common  in  research  projects.  Further 
more,  in  several  rather  successful  courseware  development 
projects,  the  leader  acted  as  instructional  designer  for  the 
group.  We  would  suggest  this  combination  be  employed  in 
those  instances  where  an  instructional  designer  has  exhibited 
managerial  skills  or  vice-versa. 

The  curriculum  coordination  is  often  performed  by  an 
especially  experienced  subject  matter  expert.  Experience  and 
breadth  of  knowledge  are  more  important  than  depth,  however. 

Instructor  experience  is  generally  accompanied  by  sub- 
ject matter  expertise.  However,  military  training  instruc- 
tors are  not  so  likely  to  be  true  subject  matter  experts. 

The  lesson  plan  and  training  materials  are  often  sufficiently 
specific  so  as  to  allow  persons  with  narrow  subject  matter 
expertise  to  be  effective  as  military  classroom  instructors, 
but  they  may  not  be  able  to  perform  more  general  roles  such 
as  a reviewer  for  a CBE  development  site.  These  instructors 
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must  be  supplemented  by  "genuine"  subject  matter  experts; 
they  should  definitely  not  be  expected  to  supervise  the 
curriculum  development. 
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TEAM  AUTHORING 


One  of  the  decisions  that  will  affect  staff  selection 
is  the  organizational  structure  of  the  staff.  At  a general- 
support  site,  all  the  authoring  may  be  done  independently, 
but  most  project  sites  adopt  some  sort  of  "team  approach." 
The  concept  of  an  authoring  team  varies  from  site  to  site: 
at  least  three  different  things  are  meant  by  a "team." 

1.  Tn  the  loosest  sense,  any  group  of  several  authors 
working  toward  the  same  set  of  goals  and  helping 

each  other  may  be  said  to  comprise  a team.  However, 
unless  they  share  responsibilities  for  programming, 
instructional  design,  etc.,  we  classify  them  as 
independent  authors  rather  than  as  members  of  a team 
<3C3- 1>. 

2.  At  the  next  level  of  team  association,  a single 
staff  member  makes  the  decisions  about  the  design, 
content,  and  programming,  but  "subcontracts"  site 
support  staff  to  furnish  the  content  and/or  the 
programming . 

3.  The  strictest  team  approach  divides  the  tasks  of 
designing,  furnishing  content,  and  programming  so 
that  no  single  person  makes  decisions  about  a]l 
three  of  these  areas.  This  approach  may  be  compared 
to  an  assembly  line  where  once  a lesson  has  completed 
one  stage  of  development,  the  control  and  responsi- 
bility passes  to  the  developers  of  the  next  stage. 

In  practice,  loops  built  into  the  process  give 
development  groups  a "second  chance"  to  see  and 
modify  the  lesson. 
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The  first  approach  requires  the  least  coordination  and 
allows  the  most  flexibility  for  staff  to  enter  or  leave  the 
project.  However,  because  few  authors  possess  all  of  the 
ten  or  so  skills  needed  for  effective  CBE  development,  there 
is  the  potential  danger  for  creating  an  uneven,  inconsistent 
set  of  lessons. 

For  that  reason,  many  sites  starting  out  with  the  first 
approach  described  above  modify  it  to  become  more  like  the 
second  approach.  Groups  of  two  to  four  staff  members  are 
placed  together  so  that  the  strengths  and  weaknesses  in  the 
various  skills  can  be  compensated  for.  In  general,  the 
result  is  enhanced  productivity  and  greater  uniformity.  In 
some  cases,  authors  who  have  been  successfully  using  the 
independent  approach  are  allowed  to  continue  in  that  mode. 

Management  needs  for  the  small-group  teams  are  somewhat 
greater  because  of  the  additional  effort  to  organize  and 
coordinate  the  teams.  Generally,  however,  it  is  the 
manager's  decision  to  assume  a larger  administrative  burden 
in  order  to  achieve  greater  control  of  the  project  <3R2.1, 
3C3.3,  3C4.1,  3C'4.?>. 

In  a team  authoring  structure  the  productivity  of 
individual  members  is  more  difficult  to  measure  and  the  loss 
of  one  staff  member  affects  the  whole  team.  On  the  other 
hand,  because  the  team  members  need  not  possess  so  broad  a 
range  of  skills,  they  are  more  easily  replaced. 

Our  experience  with  the  strictest  form  of  a team  approach 
indicates  that  for  authors  familiar  with  other  authoring  modes, 
the  lack  of  identification  with  a lesson  seems  to  offer 
fewer  internal  rewards.  However,  from  an  administrator's 
point  of  view  this  reduction  of  ego  involvement  may  be  a 
desireable  outcome  (Francis,  1976,  p.  32).  The  most  frequent 
implementation  of  the  "sub-contract"  form  of  team  authoring 
occurs  when  subject  matter  experts  plan  and  design  a lesson. 
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but  delegate  the  programming  responsibilities  to  someone 
else . 

Evaluation  of  the  efficiency  and  effectiveness  of  each 
of  these  authoring  approaches  is  confounded  by  effects  of 
the  quality  of  the  authors  and  the  objectives  of  the  site. 
Highly-skilled  authors  tend  to  prefer  and  perform  best  in  an 
independent  authoring  mode.  Generally,  when  this  mode  has 
been  tried  and  discarded,  it  was  because  the  authors’  inexper- 
ience precluded  their  writing  lessons  with  sufficient  quality 
or  in  sufficient  quantity  to  meet  project  goals.  Reduced 
author  independence  has  typically  been  viewed  as  a technique 
for  increasing  the  quality  of  the  average  lesson  and/or 
increasing  the  production  rate. 

A rather  extensive  discussion  of  authors,  teams,  and 
the  dynamics  of  authoring  structures  found  on  PLATO  has  been 
prepared  by  Mahler  et  al . (1976). 
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STAFF  SFLECTION  GUIDELINES 

In  this  section,  we  offer  some  comments  concerning  the 
selection  of  key  staff  members.  The  roles  of  director, 
instructional  designer,  and  author  are  discussed,  and  brief 
comments  are  made  about  several  other  positions. 

Director 

As  is  typically  the  case,  the  choice  of  director  is 
critical  to  the  success  of  the  project.  Therefore  this 
position  is  discussed  in  the  greatest  detail. 

In  general  the  following  have  been  found  to  be 
true:  (a)  a directorship  typically  cannot  be  handled  part 
time  when  the  staff  numbers  three  or  more,  (b)  the  director 
must  have  sufficient  time  available  to  be  able  to  watch 
students  use  CBE  lessons,  (c)  the  director  must  be  open- 
minded  enough  to  ignore  preconceptions  and  intuitions  gained 
in  other  media  or  with  other  student  populations,  and  (d)  the 
personal  interests  and  background  of  the  director  must  be 
matched  with  site  goals  (e.g.,  don't  hire  an  instructional 
designer  for  a hardware  project  and  vice-versa)  <2B1.1,  2B1.2, 
3A1 .3>. 

A frequently-overlooked  problem  is  the  question  of  the 
"distance"  of  the  director  or  other  higher  administrator  from 
the  project.  The  project  director  must  be  far  enough  away 
from  the  day-to-day  problems  of  authoring  so  that  these 
matters  don't  obscure  the  long  range  view.  Some  directors 
have  tried  to  add  directorship  responsibilities  to  their  full- 
time author  duties  <3A1.1,  3A1.2>.  At  the  other  extreme, 
some  directors  whose  jobs  entailed  non-CBF  duties  have  been 
frustrated  when  they  are  circumvented  in  the  decision-making 
process  because  they  had  too  little  time  to  familiarize 
themselves  with  the  current  problems  and  potential  solutions. 
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when  a project-oriented  site  is  to  be  started,  some 
additional  insights  can  be  offered.  It  is  most  important  to 
obtain  the  director  as  early  as  possible — as  much  as  6-12 
months  before  the  arrival  of  the  full  complement  of  staff  and 
terminals.  This  is  especially  important  if  the  director  is 
not  familiar  with  instructional  development  in  CBE.  An 
"early  start"  has  given  many  directors  a chance  to  gain 
experience  and  make  a few  mistakes  without  incurring  large 
penalties.  If  the  director  cannot  be  acquired  early,  he 
should  be  available  at  least  at  the  beginning  of  the  project; 
those  projects  which  had  no  leader  or  a known-to-be-temporary 
leader  have  tended  to  founder  until  the  director  was  appoint- 
ed <2B1.1>.  In  order  to  be  sure  that  the  project  objectives 
are  well-understood,  it  is  highly  desirable  that  the  project 
director  participate  in  the  goal  specification  phase  of  the 
project . 

At  a general-support  site,  the  director  need  not 
exhibit  the  leadership,  curriculum  development,  or  subject 
matter  skills  that  his  counterpart  in  a direct-support  site 
must.  Wowever , both  share  a need  for  managerial,  CBE, 
instructional  design,  and  media  selection  expertise  <2B2.1, 
2B2.2,  2B2.1,  2B2.«,  2B2.5>. 

Inst ruction al  Designer 

The  role  and  responsibilities  of  an  instructional 
designer  are  neither  well-known  nor  well-defined.  Even 
practitioners  may  find  it  hard  to  express  concisely  what  they 
do.  Therefore  let  us  look  at  several  aspects  of  instructional 
design  in  an  attempt  to  define  it  by  interpolation. 

The  ancestors  of  instructional  design  (and  the  sources 
of  the  jargon  and  principles  used  in  it)  are: 

Programmed  Instruction 

Bystems  Analysis 

Educational  Psychology--especially  learning  theory 

Educational  Technology  (audio/visual  equipment,  etc.) 
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Hence,  one  nay  expect  to  find  one  or  more  of  these 
skills  in  the  vita  of  an  applicant  for  an  instructional 
desipn  position. 

There  are  several  things  that  instructional  design 
sounds  similar  to,  but  is  distinct  from: 

1.  Curriculum  development  may  include  instructional 
design,  but  is  often  more  oriented  towards  the  content 
being  delivered  than  towards  the  strategems  or  techniques 
for  delivery. 

2.  Educational  psychology  research  provides  the  basis 
for  much  of  what  an  instructional  designer  does,  but  the 
instructional  designer  is  oriented  more  toward  applica- 
tions than  toward  formal  research.  Because  CBE  employs 
new  and  unexplored  technologies,  however,  instructional 
designers  may  well  turn  up  valuable  research  data. 

Staff  with  broad  instructional  design  skills  may  be 
hard  to  find.  Some  who  have  worked  only  in  another  medium 
(e.g.,  programmed  texts)  may  apply  their  experience  immedi- 
ately and  directly,  but  be  slow  to  adapt  or  develop  techniques 
effective  for  CBE. 

Author 


In  these  guidelines  we  have  used  the  term  "author"  to 
specify  the  individual  who  bears  the  largest  personal  responsi- 
bility for  the  development  of  a lesson.  The  term  "author" 
has  been  applied  to  individuals  in  so  many  different  roles 
and  with  so  vastly  different  skills  that  trying  to  establish 
a firmer  definition  of  what  an  "author"  is  does  not  seem  to 
be  a profitable  pastime.  The  next  section  describes  several 
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typical  sources  for  authors,  the  advantages  and  limitations 
of  each,  and  their  availability  and/or  ease  of  recruitment 
<3B1 . 1 >. 

Many  subject-matter  experts  become  authors  by  learning 
the  CBE  language  and  beginning  to  prepare  lesson  material. 
Because  becoming  this  type  of  author  requires  only  the  qualifi- 
cation of  subject  matter  expertise,  such  staff  are  compara- 
tively easy  to  find.  However,  SMEs  suffer  from  at  least  two 
shortcomings.  Some  of  these  author  recruits  find  it  difficult 
to  learn  to  program  quickly  and  efficiently.  Others,  lacking 
experience  with  teaching  students  or  with  the  development  of 
instructional  materials,  may  create  lessons  which  tell  rather 
than  teach  (e.g.,  their  lessons  are  non-interactive  page 
turning).  They  may  have  to  rediscover  for  themselves  the 
principles  of  effective,  efficient  instruction  <2C1.1>. 

Some  authors  are  instructors  who  have  been  taken  from 
the  classroom  and  taught  the  CBE  language.  Equipped  with 
knowledge  about  teaching  and  about  the  student  population, 
they  are  more  valuable  and  perhaps  more  expensive  than  SMEs. 

For  example,  "robbing"  the  classroom  of  its  experienced  staff 
may  not  increase  financial  costs  but  will  exact  other  costs 
in  other  ways.  Former  instructors  share  many  of  the  poten- 
tial problems  of  the  subject  matter  experts  discussed 
above.  However,  their  familiarity  with  the  students  is  an 
asset  that  will  allow  them  to  avoid  some  of  the  difficulties. 
Authors  who  retain  some  traditional  instructor  duties  may 
receive  valuable  information  and  insight  from  their  dual  role 
<2C1.2,  2C3.1>. 

Authors  who  are  primarily  programmers  can  learn  the  CBE 
language  rapidly  and  produce  their  first  products  well  ahead 
of  other  kinds  of  author  staffs.  They  may  have  a different 
under-standing  of  what  a "finished"  lesson  is,  however,  since 
revision  of  programming  for  esthetic  or  educational  purposes 
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is  not  a familiar  concept.  Though  it  is  common  to  teach  the 
CBE  language  to  subject  matter  experts  and  instructors,  it  is 
unusual  to  teach  programmers  the  subject  matter  or  instruc- 
tional design  (except  when  the  skills  are  rather  basic  and 
can  be  quickly  taught).  Thus  programmers  have  this  funda- 
mental weakness:  they  tend  to  design  well-written  programs 
which  fail  to  teach  effectively.  Conscientious  programmers 
may  find  they  have  to  spend  a large  part  of  their  time  con- 
sulting with  SMFs  or  instructors,  rather  than  programming 
<2C3.?>. 

It  is  unusual  to  use  evaluators  or  instructional 
designers  as  authors.  They  are  ordinarily  too  hard  to  find 
and  too  expensive  for  this  purpose. 

Lest  the  reader  come  to  the  conclusion  that  effective 
authors  cannot  be  found,  we  should  point  out  that  most  of  the 
shortcomings  of  the  various  types  of  authors  listed  above  can 
be  eliminated  or  alleviated  by  selecting  a site  staff  with 
balanced  abilities  and  a director  who  can  coordinate  the 
sharing  of  these  talents.  The  purpose  of  this  section  is  to 
make  a project  director  aware  of  potential  staff  weaknesses. 

Othe r Posi tj, o n_s 

number  of  programmers  needed  depends 
on  the  degree  to  which  authors  are  expected  to  program  their 
own  materials.  In  the  case  of  highly-paid  professionals 
(e.g.,  M.D.'s,  lawyers)  or  manually-oriented  technicians 
(e.g.,  machinists)  it  is  probably  more  cost-effective  to 
consider  employing  programmers  rather  than  training  the 
SMEs  to  become  proficient  programmers.  At  the  beginning  of 
any  new  project,  when  the  staff  is  new  and  inexperienced, 
there  is  often  a great  need  for  programming  skill.  As  the 
project  or  group  ages,  the  ratio  of  programmers  to  subject 
matter  experts  may  decrease  (although  the  total  site 
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expertise  in  prop^ramming  has  increased).  When  SMEs  do  none 
of  the  first  draft  programming,  a ratio  of  two  or  three 
programmers  per  SMF  has  been  suggested  by  several  project 
directors  <3B1.2,  3B1.3,  3B1.a,  3B2.2>. 

Proctor . Even  when  a project  is  too  small  to  justify 
such  a position,  the  responsibility  for  this  duty  should  be 
unambiguously  assigned.  For  example,  rather  than  stating 
that  any  staff  member  present  in  the  room  is  to  act  as  a 
proctor,  designate  one  staff  member  for  specific  days  of  the 
week  or  hours  of  the  day. 

Subj ect  matter  expert  reviewer.  Persuading  non-CBE 
staff  to  act  as  reviewers  can  bring  many  benefits.  Liaison 
is  facilitated,  rumors  are  squelched,  lessons  are  improved, 
and  a feeling  of  working  toward  and  meeting  common  goals  is 
promoted  . 

Mrtor^.  Though  this  may  seem  a rather  small  function, 
experience  shows  that  unless  the  responsibility  is  assigned 
to  someone,  small  errors  will  greatly  detract  from  otherwise 
highly-respectable  lessons. 

General  Comment 

Although  this  section  hopefully  provides  site  managers 
with  some  guidance  toward  selecting  a staff,  it  is  very 
probable  that  one  or  more  staff  members  will  be  selected  who 
are  rather  incompatible  with  the  job  responsibilities.  This 
is  especially  likely  if  the  staf^  members  are  new  to  CBE. 

One  should  anticipate  this  problem  and  consider  how  to  solve 
or  alleviate  it.  Because  new  authors'  abilities  with  the  CBF 
language  and  with  instructional  design  are  generally  obvious 
during  training,  it  may  be  prudent  to  train  more  authors  than 
will  ultimately  be  needed.  Note  that  the  final  selection 
must  be  carefully  handled  to  avoid  bad  feelings  about  not 
being  chosen. 
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A selection  process  which  may  avoid  attaching  a stigma 
to  people  not  selected  involves  offering  the  training  to  all 
members  of  an  institution.  Special  invitations  and  encourage- 
ment can  be  given  to  people  whose  backgrounds  are  promising. 

The  trainees  who  learn  quickly  and  show  early  potential  to  be 
good  authors  can  be  offered  to  be  supported  by  the  CGF  project. 
Those  performing  less  well  can  be  allowed  to  continue  without 
support  while  working  toward  their  own  teaching  goals,  Fven 
though  many  people  are  trained  and  only  a few  unsupported 
trainees  will  continue,  the  CBE  training  serves  to  familiar- 
ize the  staff  of  an  institution  with  the  techniques  and 
objectives  of  CBE. 
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FRACTIOrjAL  APPOINTMEf'TS 

This  section  could  be  summarized  in  one  sentence:  there 
have  been  too  many  full  time  authors  and  not  enough  full  time 
directors.  Mow  for  an  expansion  of  this  terse  statement. 

Authors . If  current  staff  members  are  to  be  selected 
and  transferred  to  a CBF  group,  it  often  proves  effective 
to  let  them  keep  a fractional  commitment  to  their  former 
position.  When  course  improvement  via  the  introduction  of 
CPF  is  the  objective,  such  a staff  member  can  act  as  a bridge 
between  current  instructors  and  the  CPE  group.  Tf  an  instruc- 
tor is  involved  part  time  with  CRE,  he  can  keep  abreast  of 
developments  in  the  course  and  likewise  build  student  enthu- 
siasm and  find  volunteers  for  lesson  tryouts.  Depending  on 
the  potential  need  for  liaison  and  abilities  of  an  instructor 
toward  that  end,  it  may  be  advisable  to  carry  his  full  salary 
but  "release"  him  to  work  part  time  in  the  course.  Contin- 
ued involvement  and  commitment  to  the  course  are  helpful  for 
choosing  appropriate  topics  for  CRE,  maintaining  rapport  with 
the  rest  of  the  instructors  and  squelching  any  unpleasant 
rumors  which  arise.  Some  authors  have  wished  that  they  had 
duties  in  addition  to  those  of  writing  CRE  lessons  because 
they  felt  the  need  for  other  stimulation  in  order  to  be 
creative. 

Split  appointments  are  not  without  substantial  problems, 
however.  Recause  CRE  development  is  often  less  structured 
and  deadlines  are  less  firm,  staff  with  split  appointments 
often  find  their  teaching  or  other  duties  encroaching  on 
their  CRE  time.  That  is,  tests  must  be  written  and  lab  books 
must  be  graded.  But  a CRE  lesson  revision  may  be  postponed 
or  skipped,  and  data  analysis  may  be  ignored  or  done  in  a cur- 
sory manner.  Therefore,  it  is  advisable  to  set  aside  and 
enforce  separate  time  slots  for  each  kind  of  endeavor.  Sepa- 
rate office  locations  are  often  an  aid. 
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When  a curriculum  is  being  rapidly  developed  by  a large 
group,  the  time  needed  for  coordination  within  the  group  is 
typically  so  large  that  an  author  with  less  than  a halftime 
appointment  makes  little  progress.  (This  is  not  true  for 
support  staff  who  need  not  attend  all  staff  meetings.)  When 
independent  modules  are  being  constructed  by  an  instructor 
for  his  own  course,  a quarter-  or  third-time  appointment  is 
minimal.  The  topic  of  what  fraction  of  time  is  optimal  for 
various  staff  positions  is  treated  at  considerable  length 
in  <2CU.1,  2C4.2,  2CJ4.3,  2C5.1,  2C5.2,  2C5.3>. 

Directors . Because  good  administrative  staff  are  rare 
and  expensive,  there  is  a tendency  to  spread  their  talents 
too  thin.  The  director  ought  to  have  time  to  regularly 
see  students  and  staff  working.  There  should  be  a staff 
member  (called  director,  manager,  supervisor,  coordinator, 
whatever)  who  oversees  day-to-day  problems.  If  there  are 
more  than  about  3 FTFs  or  5 people,  that  person  should  be 
full  time,  not  half  time  <2R3.1,  2B3.2>. 
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TOUCHSTONFS  '='OR  STAFF  SFLFCTIOr.' 


Gereral  education  as  evidenced  by  academic  degrees  or  other 
awards  seems  to  be  helpful.  A firm  math  background  makes 
learning  the  CFE  language  easier. 


Persons  who  have  recently  been  enrolled  in  some  sort  of 
formal  education  classes  tend  to  more  quickly  learn  about 
C°F  and  how  to  operate  the  terminal. 


'■’aster  teachers  with  ?0  or  more  years  of  experience  have 
been  found  to  be  invaluable  sources  of  information  about 
students,  their  problems,  criso  examples,  scenarios,  etc. 
but  many  of  them  find  it  difficult  to  learn  to  program. 

Instead  of  expecting  them  to  become  proficient  in  the  CPF 
]anguage,  one  can  exploit  their  knowledme  to  organize  J 

content,  suggest  lesson  strateeies  and  review  the  final  ;| 

product.  I 


A greatly-needed  personality  characteristic  is  the  ability 

to  take  criticism  and  to  learn  thereby--a  sort  of  humility. 

■’"he  staff  member  who  portrays  a "the  customer  is  always 

) 

right"  attitude  and  encourages  constructive  criticism  is  one 

to  choose.  The  presence  of  such  an  attitude  can  be  judged 

from  the  attitudes  and  practices  such  as  the  following: 

Poes  the  candidate  submit  only  final -typed  versions  of 
his  papers  and  memos  for  critiquing  by  colleagues  (and 
thereby  suggest  that  comments  are  not  really  wanted)? 
Poes  the  candidate  request  critiques  at  all?  Are  revis- 
ions made  on  the  basis  of  them?  Do  student  comments  on 
te.st  items  cause  improvements  to  be  made?  Are  student 
comments  taken  seriously'’  As  an  instructor,  has  the 
candidate  ever  sought  opinions  of  others  about  how  to 
teach  or  test  a topic?  hoes  the  candidate  regularly 
revise  classroom  handouts? 


93 


Experience  in  teaching  is  naturally  a valuable  asset.  Note 
that  experience  in  an  individualized  teaching  format  (e.g., 
tutoring,  TV  tapes,  programmed  instruction)  is  more  valuable 
than  ordinary  classroom  experience  (Avner,  personal  communi- 
cation) . 

Preparing  a CRE  lesson  requires  a great  deal  of  technical 
writing.  A candidate's  skills  may  be  assessed  by  examining 
books  and  articles.  Letters,  tests,  and  directions  for  lab 
exercises,  for  example,  should  also  be  examined  because  they 
may  be  judged  more  easily  without  reference  to  their  subject 
matter  merit  and  are  more  likely  to  reflect  the  ability  of 
the  candidate  unaided  by  editors.  Of  primary  importance  is 
clarity;  next  is  succinctness.  Consider  whether  the  candi- 
date seems  to  be  striving  to  impress  his  audience  or  to 
communicate . 

Specifying  the  level  of  subject  matter  expertise  is  often 
difficult.  A useful  rule  of  thumb  (Avner,  personal 
communication)  is  that  the  lesson  author  should  be  "one 
level  higher"  than  the  students.  Thus  a person  who  just 
completed  a course  is  not  a good  choice  for  authoring  a 
lesson  for  that  course.  A recent  student  can  offer  very 
valuable  insights,  but  not  in  the  role  of  the  author. 

As  a further  guide  toward  determining  what  "one  level" 
means,  consider  that  high  school  students  are  considered 
able  to  tutor  grade  school  .students,  college  students  able 
to  teach  high  school  students.  In  cases  where  degrees  do  not 
apply,  an  estimate  of  two  to  three  years  of  relevant  field 
experience  (not  merely  two  to  three  years  since  training) 
may  be  equivalent.  The  topic  of  students  as  authors  is 
treated  in  the  next  section. 
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MISCELLAMFOL'S  CONSIDER ATIOflS 

Students  as  authors.  In  many  academic  situations,  one 
finds  a talented  pool  of  relatively  inexpensive  labor  in  the 
students  of  the  institution.  For  a variety  of  reasons,  the 
students  often  adapt  more  quickly  to  the  learning  of  new 
languages  and  the  use  of  the  CPE  system  than  do  the  faculty. 

In  such  a situation  someone  frequently  suggests,  "Why  not 

let  the  students  write  the  lessons?"  j 

If  programming  CBE  lessons  were  identical  to  writing  | 

CBE  lessons,  there  would  be  no  problem.  However,  unless  1 

the  students  have  sufficiently  more  background  than  the  | 

course  they  are  writing  for,  they  are  likely  to  have  diffi- 
culty organizing  the  material  in  any  way  different  from  that 
in  which  they  were  taught.  Thus  juniors  and  seniors  might 
write  freshman/ sophomore  level  lessons,  but  faculty  should 
write  lessons  for  the  juniors  and  seniors.  Cince  most  stu- 
dents are  available  half  time  or  less,  the  problems  noted 

previously  for  part-time  work  are  present.  Furthermore,  the  ^ 

previous  suggestions  about  the  advantages  of  part-time 
employees  assumed  that  the  rest  of  the  employee's  time  was 
spent  in  an  activity  which  complemented  the  authoring  tasks; 
this  is  unlikely  to  be  true  for  a student.  Finally,  exper- 
tise built  up  in  students  will  be  lost  by  graduations 
unless  it  can  be  transferred  to  other  students  or  permanent 

I 

staff  <2C1.3,  PC1.4,  ?C1.3.  3C3.1>.  j 

On  the  bright  side,  the  low  cost  of  student  labor,  plus  ; 

its  availability  may  make  so  large  a payoff  that  the  risks  ' 

are  reasonable.  It  will  no  doubt  he  a valuable  learning 
experience  for  the  student,  and  a student  may  be  able  to 
introduce  and  teach  faculty  about  programming  more  quickly 
and  easily  than  otherwise  possible.  However,  student-written 
lessons  will  probably  need  extensive  revisions  before  they 
are  ready  for  classroom  use. 
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Students'  abilities  to  rapidly  acquire  proficiency  with 
computer  lanRuapes  provide  a resource  to  be  tapped  in  two 
ways.  First,  students  can  be  nicely  used  as  programmers  in 
on-site  projects.  Second,  students  can  kindle  the  interest 
of  their  instructors  and  thereby  start  new  projects.  For 
example,  several  university  sites  which  require  every  student 
programmer  to  have  a faculty  sponsor  have  found  that  a major- 
ity of  their  new  faculty  users  are  people  whose  interest 
was  aroused  by  being  a sponsor  <332. ?>.  One  project  which 
paired  students  with  professors  to  produce  lessons  was  able 
to  achieve  very  low  production  costs  and  some  remarkably  good 
lessons  (Grimes,  1975). 

Matching rank  and  role.  It  is  important  to  match  the 

rank,  position,  and  salaries  of  staff  members  to  their  roles — 
especially  for  support  staff.  For  example,  when  instructional 
designers  or  programmers  are  held  in  low  esteem  by  virtue  of 
their  lack  of  formal  "standing,"  their  inputs  (no  matter  how 
valuable)  tend  to  be  brushed  aside.  This  situation  can  arise 
easily  when  the  subject  matter  experts  are  expensive  profess- 
ionals such  as  lawyers,  M.D.s,  etc.  Neither  full  professors 
nor  four-star  generals  have  the  breadth  of  knowledge  to 
ignore  comments  from  support  staff.  Unfortunately,  they  may 
have  authority  to  do  so  <2C2.'1>. 

If  possible,  the  most  experienced  and  qualified  instruc- 
tional designers  or  evaluators  should  have  a higher  rank  than 
the  authors.  In  several  projects  where  this  hierarchy  was 
reversed,  the  effect  was  detrimental. 
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SUGGESTIONS  FOR  AN  AUTHOR  TRAINING  PROGRAM  j 

Give  the  new  author  a student  signon  to  a list  of  lessons  | 

which  must  be  completed  with  a passing  score.  Make  sure  the  I 

list  includes  both  good  and  ooor  examples  of  CBE.  This  is  ! 

a first  step  for  promoting  the  author's  empathy  for  the 

student.  ; 

If  possible,  pair  up  a new  trainee  with  an  experienced 
and  successful  author.  If  no  "old  hands"  are  available, 
pair  up  two  new  authors  and/or  a non-CBE  colleague.  Set  up  a 
schedule  so  that  the  group  meets  two  to  three  times  per  week 
to  go  over  progress,  problems,  and  ideas.  Verify  that  meet- 
ings take  place  for  the  first  several  months.  The  important 
point  here  is  that  the  author  grows  accustomed  to  regularly 
showing  and  sharing  his  plans  and  products  with  others.  This 
series  of  planned  meetings  will  hopefully  avoid  the  "hermit 
author"  phenomenon. 

Advise  new  authors  to  expect  that  they  will  have  to  try  their 
lesson  materials  on  students  at  early  and  regular  intervals. 

The  first  trial  should  come  no  later  than  eight  weeks  after 
the  beginning  of  training.  Authors  cannot  restrict  their 
learning  to  formal  sources  such  as  manuals,  consultants,  and 
research  reports.  They  must  also  learn  from  contact  with 
students.  Encourage  or  force  authors  to  test  incomplete  seg- 
ments of  lessons. 

Insist  that  the  first  lesson  be  "finished"  or  set  aside  and 
the  second  lesson  started  after  two  to  three  months.  Encour- 
age authors  to  prepare  written  (especially  on-line)  notes  and 
comments  as  part  of  their  preparation  for  reviewing  other 
authors'  lessons. 
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Have  the  instructional  designer  work  with  every  new  author  at 
least  once  a week.  When  student  testing  begins,  the  evalua- 
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Provide  the  new  author  with  a repertoire  of  known,  tested 
strategies  and  instructional  design  approaches.  Such  a 
repertoire  can  allow  a novice  to  develop  high  quality  lessons 
rather  quickly. 

If  training  can  he  given  at  a major  CBE  center,  make  sure 
that:  authors  visit  operating  classrooms;  staff  meet  poten- 

tial on-line  colleagues,  reviewers,  etc.;  staff  meet  various 
system  consultants,  maintenance  staff,  business  office  staff, 
evaluators,  etc.;  author^,  instructional  designers,  and 
evaluators  visit  the  CBE  center's  library  to  find  books  and 
journal  suggestions  for  their  home  library. 

Concentrate  the  introduction  to  CBE  and  language  training  in 
as  short  amount  of  time  as  possible.  Although  it  is  possible 
to  teach  about  CBE  at  a rate  of  two  to  three  hours  per  week, 
it  is  far  more  efficient  to  schedule  a concentrated  course 
during  a vacation  period  or  even  a marathon  weekend  (with 
follow-up  training  at  a slower  pace). 

It  is  preferable  that  project  objectives,  content  decisions, 
responsibilities,  course  revision  plans,  etc.  be  settled 
completely  before  training  starts.  The  period  after  initial 
training  is  a good  time  to  review  such  decisions,  but  too 
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much  time  spent  then  d'.'lls  enthusiasm  and  allows  forgetting 
of  recently-learned  programming  skills. 

If  staff  experienced  in  CBE  can  be  employed  as  consultants, 
the  best  time  to  have  them  is  during  planning  period.  If 
they  are  familiar  with  the  specific  CBE  system  to  be  implemen- 
ted, they  should  be  employed  during  the  first  weeks  following 
training. 

If  some  staff  arrive  early  or  have  free  time  prior  to  train- 
ng  (and  access  to  CBE  terminals),  they  should  be  directed  to 
prepared  written  reviews  of  existing  materials  in  their  con- 
tent field.  These  should  not  be  done  from  a "browsing"  or 
"can-I-use-it"  perspective,  but  rather  as  critiques  of  good 
and  bad  styles,  things  to  emulate  and  things  to  avoid. 

If  the  staff  is  ccmposed  of  members  who  retain  their  home 
departmental  affiliations,  efforts  to  foster  a group  spirit 
are  useful.  Seminars  with  guest  speakers,  luncheons  etc., 
might  be  considered. 

If  high  quality  lessons  written  at  another  site  wo’orld.  be 
useful  for  students  in  the  target  course(s),  trv  to  begin 
to  use  them  as  soon  as  possible.  By  using  well-tested 
lessons  from  another  site,  new  authors  will  get  a feeling 
for  how  a class  should  react  to  their  lessons  after  they  are 
pol ished . 

Note:  Additional  specific  recommendations  for  author  training 
on  the  PLATO  IV  system  are  found  in  Francis  (1976). 


PART  III-RECOPIHEUDATIONS  FOR  MANAGING  A CBE  SITE 
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MANAGING  FORMATIVE!  D>^VEI,OPMENT  (PRODUCTION  STEPS) 

The  following  are  commonly-observed  problems  at  CBE 
development  sites. 

Colleague  reviews.  As  noted  by  Francis,  Goldstein,  A 
Call-Himwich  (1975)  and  Call-Himwich  (1d76),  authors  have 
tended  to  wait  too  long  before  shewing  their  lessons  to 
colleagues  for  a formal  or  informal  review.  In  some  cases, 
when  reviews  did  take  place,  problems  and  errors  noted  during 
the  review  were  nevertheless  left  unrepaired.  Both  the  ref- 
erences cited  above  contain  detailed  suggestions  for  encour- 
aging positive,  constructive  on-site  or  off-site  reviews  of 
lessons  <4B1 . 1 , 4B1.2,  4B1.3,  4B1.4>. 

Student  testing.  Left  to  their  own  instincts,  inexperi- 
enced authors  also  tend  to  wait  far  too  long  before  testing 
lessons  or  lesson  segments  on  students.  They  always  have 
"just  one  more  display"  to  finish  before  getting  student 
feedback.  Firm  managerial  control  can  avoid  the  problems 
associated  with  too  much  author  effort  without  student 
feedback  <4B2.1,  4B2.2,  4B2.4,  4B2.5,  4B2.6>. 

Student  response  data.  Most  authors  collect  sufficient 
high-quality  data  about  their  lessons  to  provide  a very 
valuable  basis  for  lesson  revision.  However,  because  some  of 
the  techniques  for  analysis  are  new  to  them  and  because  the 
storage  space  available  seldom  allows  a leisurely  investiga- 
tion of  the  data,  some  authors  adopt  a "print-and-stack" 
philosophy.  In  order  to  preserve  the  data,  they  get  printouts 
of  it,  then  delete  the  on-line  copy.  Although  this  procedure 
is  preferable  to  simply  deleting  the  data,  it  often  lulls  the 
author  into  feeling  his  job  is  done.  He  may  intend  to 
examine  the  data  someday,  but  in  fact,  that  rarely  happens. 

The  hardcopy  version  is  not  computer-searchable,  and  the 
author's  memory  and  enthusiasm  for  correcting  problems 
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diminishes  with  time.  Soon  all  he  has  is  a stack  of  paper. 
The  manager  of  a site  must  not  only  allow  time  for  examina- 
tion of  data  but  also  must  see  that  a site  staff  member 
experienced  in  using  student  response  data  guides  other 
authors  in  the  use  and  interpretation  of  their  data  (Francis 
h Weaver,  1b77)  <Jin2.3>. 

Revision . The  above  three  topics  deal  with  various 
aspects  of  obtaining  feedback  about  lessons  to  enable  the 
author  to  make  an  effective  and  efficient  revision  of  the 
lesson.  Much  of  the  effectiveness  of  CBE  is  possible 
because  of  the  rapid  and  specific  feedback  available  from 
student  tryouts.  For  strategies  and  topics  where  the  medium 
offers  no  extraordinary  advantage,  one  must  expect  that  the 
first  version  of  a lesson  will  be  no  better  (and  possibly 
poorer)  than  the  same  material  in  a conventional  medium. 
However,  by  obtaining  and  using  the  feedback  available,  an 
author  can  continually  improve  the  CPF  lesson  until  it  is 
superior  to  a similar  lesson  written  in  a medium  without 
such  thorough,  sensitive  feedback  ( Cal  l-'iimwich  h Steinberg, 
1d77)  . 

In  most  other  media,  revisions  are  difficult  to  make 
(especially  after  "publication"),  data  is  more  expensive  to 
collect  and  analyze,  and  revised  forms  appear  at  intervals 
of  four  to  six  years.  With  CBE,  each  new  class  offers  the 
potential  for  testing  a new  revised  version  with  relatively 
little  effort. 

It  may  be  helpful  to  suggest  to  authors  that  they  view 
a lesson  as  an  hypothesis  about  how  to  teach  the  material, 
hypothesis  may  be  found  false  or  in  need  of  modification 
without  criticism  of  its  formulator.  Considering  a lesson 
as  an  hypothesis  de-emphasi zes  the  "grind  'em  out,"  one-shot 
production  mode  which  some  authors  too  readily  adopt. 


An 
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Lesson  Integration.  The  formalities  of  listing  and 
testing  prerequisites,  standardizing  terminology,  cross- 
referencing,  and  coordinating  lesson  content  are  often 
perceived  to  be  rather  bothersome  and  unimportant  to  an 
author  "being  creative"  with  CBE.  However,  unless  these 
topics  are  attended  to,  the  result  will  be  difficult  to  mold 
into  a curriculum  and  will  remain  just  a collection  of  topics 
on  a subject. 

Standards . The  need  for  lesson  standards  and  lesson 
documentation  has  been  discussed  in  part  I.  Unfortunately, 
such  standards  are  frequently  outlined,  discussed,  and 
adopted  at  the  beginning  of  a project  but  ignored  ever  after. 

Production  goals.  All  the  previous  items  might  be 
solvable  if  authors  could  be  given  additional  time  to  prepare 
each  lesson.  Although  realistic  timelines  are  essential, 
both  over-generous  time  allocations  and  the  lack  of  short- 
term goals  are  a disservice  to  developers  and  a waste  of 
resources . 

CRE  suffers  from  a lack  of  reliable  production  yard- 
sticks: any  two  "topics,"  "teaching  points,"  or  "objectives" 

can  differ  widely  in  the  effort  needed  to  convert  them  into 
an  effective,  efficient  CBE  lesson.  "Student  contact  hours 
produced"  is  no  better  a measure  because  it  rewards  the 
production  of  long-winded,  inefficient  materials.  Neverthe- 
less, a manager  must  set  goals  and  deadlines  for  authors, 
estimating  the  difficulty  of  the  tasks  and  the  abilities  of 
the  author.  It  is  difficult,  but  necessary,  to  strike  a 
balance  between  a too-rigid  enforcement  of  deadlines  which 
frustrates  authors  and  lowers  quality  and  a too-relaxed 
observance  of  undefined  goals  which  may  allow  some  authors  to 
make  a career  of  a single  lesson. 

Liaison.  Liaison  problems  often  drive  authors  into 
seclusion:  it  is  easy  to  imagine  a committee  haggling  over 
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the  content  of  a lesson,  course,  etc.  Normally  an  instruc- 
tor simply  teaches  as  he  wishes  even  though  fellow  instructors 
may  teach  somewhat  differently.  This  "leave  me  alone,  so  I 
can  do  it  my  way"  approach  works  reasonably  well  in  a lecture 
format  where  the  instructors  are  competent  and  are  responsi- 
ble for  all  the  teaching.  However,  such  an  attitude  in  a 
multiple-author  project  means  the  lessons  may  not  be  coordi- 
nated; in  a one-person  project,  it  means  that  few  other 
instructors  will  ever  take  advantage  of  the  large  courseware 
development  e fort  made  by  the  author. 


i 

1 

I 

I 


104 

CLASSROOM  MANAGEMENT 


To  what  extent  will  recreational  uses  be  permitted? 


Site  directors  usually  answer  this  question:  "To  the 
extent  that  recreational  uses  don't  interfere  with  other 
uses."  The  following  suggestions  are  an  attempt  to  put  this 
general  answer  into  a specific,  practical  format  so  that 
serious  workers  taking  a break  will  not  feel  guilty,  but 
non-productive  users  will  not  compete  with  or  detract  from 
site  operations  and  goals. 

Terminals . If  there  are  recreational  users  present, 
there  should  always  be  at  least  one  additional  terminal  free 
so  that  a potential  user  does  not  have  to  dislodge  a recrea- 
tional user  and/or  wait  to  have  the  user  finish.  Because 
game  players  may  switch  to  non-recreat ional  uses  when  asked 
to  leave,  and  because  priority  users  may  leave  rather  than 
confront  a recreational  user,  the  "waSte"  of  a free  terminal 
must  be  tolerated  by  second-priority  users.  If  a site  is 
large  (more  than  12  terminals)  it  has  often  been  found 
appropriate  to  have  two  or  more  vacant  terminals  whenever 
recreational  users  are  present.  A site  policy  regarding 
terminal  use  can  be  partially  implemented  on  the  PLATO  system 
via  the  "current  usage  controller"  or  CUC. 

ECS  usage.  The  use  of  lesson  storage  space  (ECS)  can 
also  be  controlled  by  the  CUC  of  the  PLATO  system.  Users 
working  in  their  own  lesson  spaces  can,  however,  consume 
space  that  would  be  used  by  high  priority  users,  were  it 
available.  Therefore,  the  site  proctor  of  a general-service 
site  must  regularly  check  the  contents  of  unfamiliar  lessons. 
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Noise  and  clutter.  It  has  been  widely  observed  that 
recreational  users  tend  to  have  a variety  of  habits  which 
may  be  bothersome  to  other  users.  For  example,  many  games 
require  or  encourage  rapid  banging  on  keysets.  Because 
recreational  users  are  frequently  limited  to  late-night  and 
early-morning  hours,  food  wrappers,  bottles,  and  other  debris 
may  accumulate  near  the  terminals. 

Opinion  and  impression.  One  of  the  greatest  detriments 
caused  by  excessive  recreational  use  is  that  people  who  are 
casually  acquainted  with  a site  may  begin  to  associate  CBE 
with  recreational  use,  i.e.,  non-academic,  non-serious 
endeavor.  Fortunately,  this  negative  impression  is  usually 
formed  only  when  game  playing  is  excessive  (i.e.,  to  the 
practical  exclusion  of  more  legitimate  uses);  the  occasional 
use  of  recreational  programs  as  a diversion  by  hard-working 
staff  seldom  leaves  a bad  impression. 


Vfho  can  give  demonstrations?  When  may  they  be 
scheduled?  Can  site  resources  (such  as  terminals)  be 
reserved  or  given  priority  during  demonstrations? 


Because  of  the  novelty  of  CBF. , there  is  a tendency  by 
site  staff  and  higher  administrators  to  "show  it  off"  rather 
frequently*  Unless  this  tendency  is  controlled  and  managed, 
a large  amount  of  resources  can  be  spent  without  furthering 
the  objectives  of  the  project  or  site.  At  one  site  in  our 
experience,  the  entire  staff  lost  a day's  work  preparing, 
waiting  for,  and  re-hashing  the  outcome  of  each  demonstration. 
A separate  demonstration  room  with  a limited-length  standard 
demonstration  is  a partial  solution  to  this  problem. 
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MAINTAINING  STAFF  MORALE 

The  following  is  a list  of  suggestions  for  rewarding 
individual  staff  members.  It  is  often  desirable  to  grant 
small,  non-monetary  rewards  for  hard  work,  competence,  or 
other  exemplary  habits.  Like  students,  staff  members  respond 
to  positive  feedback.  The  list  below  includes  tasks  which 
provide  strong  internal  rewards  and  thus  contribute  to 
overall  Job  satisfaction  and  also  management  actions  which 
can  provide  reinforcing  feedback  to  employees. 

If  a lesson  is  working  reasonably  well,  the  author  generally 
gets  an  ego  boost  each  time  students  use  it.  The  student 
and  instructor  comments  will  always  include  a few  new  compli- 
ments and  the  (hopefully)  rather  minor  corrections  needed 
will  remind  the  author  of  just  how  polished  the  lesson  has 
become . 

The  respect  of  colleagues  is  sought  after  by  staff  in  all  1; 

fields.  The  PLATO  system  offers  an  additional  dimension:  i 

on-line  colleagues.  Because  of  the  PLATO  system’s  communica-  ; 

tion  features,  it  is  possible  to  become  acquainted  without 
ever  meeting  face-to-face.  This  allows  a relatively  inexpen- 
sive metnod  for  exchanging  professional  opinions  and  finding 

additional  users  for  courseware.  Since  it  is  a compliment  to  i 

have  one's  courseware  in  use  in  wide  circles,  the  PLATO  sys-  ; 

tern’s  geographical  dispersion  provides  a "coast-to-coast"  j 

vehicle  for  rewarding  the  author.  ^ 

I 

(' 

Recognition  of  the  contributions  of  staff  can  be  made  on  the 
title  page  of  a lesson;  in  site  brochures,  reports,  and  pro- 
posals; and  in  references  and  co-authorships  for  journal 
articles. 


* 

1. 
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For  most  staff,  it  is  a compliment  to  be  asked  to  give  a 
demonstration  of  the  CBE  system  and  the  courseware  of  the 
site.  Giving  demonstrations  (or  speeches)  or  being  a project 
representative  at  meetings  carries  a reward  in  proportion  to 
the  responsibility  the  task  requires.  A smaller  but  still 
substantial  reinforcement  is  given  when  a staff  member's 
lesson  is  used  by  someone  else  for  demonstrations. 

Lesson  reviews,  because  of  their  basic  nature  tend  to  be  dis- 
couraging to  receive.  The  most  important  way  of  reducing 
the  negative  aspects  of  lesson  reviews  is  to  avoid  giving 
an  author  a review  which  suggests  massive  revisions.  This 
can  be  done  by  providing  frequent,  careful  monitoring  of  the 
content  and  instructional  design.  Several  short  reviews 
throughout  the  development  of  the  lesson  are  more  useful  and 
more  palatable  than  a massive  review  at  the  end. 

i 

Peleasing^carce  resources  such  as  signons  and  lesson  space 
for  personal  use  by  staff  members  (or  their  families)  is  a 
substantial  reward. 

More  than  in  most  environments,  the  work  that  one  does  on  a 
CDE  system  can  be  performed  at  any  hour  of  the  day.  If  co- 
ordination requirements  are  low,  it  may  be  possible  to  offer 
extremely  flexible  working  schedules.  In  fact,  because  the 
potential  flexibility  is  obvious  to  all,  failure  to  offer 
it  may  produce  discontent. 
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EVALUATION  ISSUES 

The  two  obvious  directions  for  an  evaluation  are  a 
comparison  of  the  CBE  materials  against  a pre-established 
goal  or  their  comparison  to  an  existing  situation. 

Non-comparative  Evaluations 

The  well-known  difficulties  of  insuring  exactly  equiva- 
lent conditions  for  a "new  vs.  old"  comparison  leads  some 
people  to  make  a "new  vs.  goal"  evaluation.  That  is,  rather 
than  comparing  the  new  CBE  materials  to  the  traditional  non- 
CBE  course,  they  measure  the  extent  to  which  the  CBE  materials 
met  the  objectives  stated.  Some  people  feel  that  they  can 
thus  avoid  setting  up  a complex  research  design,  dispense 
with  the  interpretation  of  statistical  analyses,  and  generally 
eliminate  the  hypotheses,  assumptions,  control  groups,  and 
detailed  reports  associated  with  a "formal  comparative 
analysis."  They  replace  these  difficult  and  time-consuming 
procedures  with  production  or  operating  goals  and  then  merely 
determine  "if  they  made  it." 

For  example,  under  this  mode  of  operation,  objectives 
might  be  to  "produce  10  role-conflict  simulations"  or  "teach 
1 dO  students  to  use  the  Wheeler-Whitson  apparatus  each  year 
via  CBE."  The  evaluation  of  the  latter  projects  seems  simple: 
one  determines  if  the  simulations  have  been  written  or  if  the 
students  have  completed  the  lessons. 

Although  this  form  of  evaluation  may  be  appropriate 
during  the  early  stages  of  an  evaluation,  its  use  as  a sole 
summativp  measure  is  shortsighted.  One  simply  cannot  avoid 
a comparative  evaluation.  Comparisons  may  be  made  with 
little  basis  and  few  facts,  but  students,  colleagues,  and 
administrators  will  decide:  Was  it  worth  the  effort?. 

Should  we  continue?,  etc.  Skeptics  may  suggest  alternative 


uses  for  the  monetary  and  personnel  resources,  use  of  a 
different  CPF  system  or  a totally  non-CBE  medium,  adoption  of 
a different  curricular  or  instructional  design  philosophy, 
or  implementation  under  different  conditions.  For  example,  a 
site  may  "successfully”  create  its  10  simulations,  only  to 
find  them  challenged  as  unrealistic,  not  requiring  CRE,  or 
boring.  Or  the  simulations  may  be  highly  respected,  but  too 
expensive  to  support. 

Because  administrators  in  some  fields  (e.g.,  engineering) 
may  be  used  to  accepting  as  "success"  the  fact  that  something 
new  has  been  found  or  created,  they  may  allow  "meets  the 
objectives"  evaluations  initially.  However,  comparisons  with 
alternatives  are  ultimately  inevitable.  Therefore,  because 
of  this  inevitability,  project  staff  should  be  encouraged  to 
gather  information  and  prepare  an  evaluation  so  that  the 
predictable  evaluation  questions  can  be  answered  on  the  basis 
of  well-documented  facts  rather  than  on  soeculation. 

Comparative  Evaluations 

The  watchword  for  making  comparative  evaluations  is 
(what  else!)  "objectives."  Unless  you  carefully  consider 
the  objectives  of  the  evaluation,  you  may  be  saddled  with 
one  of  several  common  problems. 

The  first  problem  is  trying  to  measure  (and  compare) 
everything . As  noted  in  the  discussion  in  part  I,  there  are 
a multitude  of  comparisons  that  can  be  made.  Only  a few  are 
relevant  and  important  in  each  situation.  Rather  than 
spreading  an  evaluation  effort  too  thin  in  order  to  measure 
many  parameters,  one  should  examine  the  objectives  of  the 
evaluation  to  determine  which  measurements  are  critical, 
and  concentrate  on  those.  A set  of  contingency  plans  for 
the  evaluation  should  be  drawn  up  simultaneously. 
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The  second  problem  is  often  found  to  be  illusory 
after  it  is  investigated,  but  it  nevertheless  produces  great 
frustration  until  that  time.  Here  is  the  apparent  dilemma: 

If  you  use  CBE  to  teach  topics  in  the  way  you  used  to,  you 
haven’t  exploited  its  potential  and  it  will  probably  not  be 
cost-effective  for  most  applications.  However,  in  this 
situation  the  testing  is  comparatively  simple:  you  can  use 
the  same  tests  that  were  used  before  for  CRE-taught  students. 
On  the  other  hand,  if  you  use  CBF  lessons  which  teach  at  a 
higher  cognitive  level  via  simulations,  modeling,  role- 
playing  and  other  techniques  which  require  a computer,  you 
have  two  choices  with  respect  to  testing.  If  you  use  the  old 
pre-CBE  tests,  they  will  be  a fair  comparison  for  the 
original  topics,  but  will  not  indicate  that  students  taught 
via  CBE  may  have  learned  new,  high-level  skills.  If  you 
write  a new  test  which  measures  the  high-level  or  CBE-taught 
skills,  it  will  not  be  possible  to  compare  results  from  the 
two  different  tests.  It  is  not  fair  to  give  the  new  test  to 
students  taught  without  the  CBE  lessons  because  the  CRE- 
taught  students  would  have  an  unfair  advantage. 

The  way  out  of  the  above  dilemma  is  (naturally)  to 
re-examine  the  project  objectives  to  answer  the  following 
question:  "Are  the  old  objectives  satisfactory  as  they  stand 

(and  are  you  thus  looking  for  an  improvement  in  teaching 
techniques,  motivation,  retention,  etc.)  or  are  the  objectives 
inadequate  or  obsolete?  If  the  old  tests  i.re  good  tests  of 
the  objectives  and  are  reliable,  they  should  be  kept.  If  the 
objectives  have  changed  or  if  the  old  exams  do  not  adequately 
measure  them,  test  development  is  needed. 

Evaluating  "CBE" 

It  should  be  noted  that  it  is  very  difficult  to  "test 
the  effectiveness  of  CBE  itself"  apart  from  the  lessons, 
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strategies,  and  implementation  of  the  CBF  system.  This 
is  another  way  of  saying  that  CBE  is  a medium,  not  a treat- 
ment. One  can  evaluate  the  ability  of  a CBE  system  to  allow 
various  teaching  strategies  and  kinds  of  lessons,  and  to 
facilitate  particular  implementations,  but  most  other  evalu- 
ation questions  depend  on  the  way  in  which  the  lessons  were 
written  or  implemented.  Thus  the  question,  "Did  the  CBE 
system  increase  the  student  scores?"  should  be  rephrased, 

"Did  the  lessons  written  for  the  CBE  system  increase  student 
scores?"  As  reformulated,  the  question  suggests  the  possi- 
bility that  another  set  of  lessons  might  have  done  a better 
or  poorer  job  of  increasing  student  scores.  It  thus  clarifies 
the  focus  of  the  evaluation  and  hopefully  prevents  post  hoc 
speculation  about  whether  the  results  were  obtained  because 
of  or  in  spite  of  the  CBE  implementation. 
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THE  PROJECT  MANAGER'S  LIST  OF  WARNING  SIGNALS 

The  director  of  a CBE  pro.ject  and  the  supervisor  of  such 
a director  should  both  be  alert  for  "smoke"  that  indicates 
smoldering  "fires"  within  a CBE  development  effort.  The  list 
assembled  below  is  not  complete,  but  is  useful  both  as  it 
stands  and  because  it  suggests  other  danger  signals. 

If  touch  input,  microfiche,  and  audio  output  are  incorporated 
into  lessons,  but  are  not  used  when  testing  the  student,  their 
use  may  be  trivial  and  decorative  only.  If  these  features 
are  needed  to  teach,  they  are  normally  also  needed  for  test- 
ing. Similarly,  lessons  which  use  animations,  simulations, 
and  case  studies  to  teach  probably  need  these  techniques  to 
test.  If  they  do  not,  a brief  investigation  may  be 
warranted . 

If  microfiche  images  and  drawings  are  presented  with  no 
questions  to  check  the  students  understanding  of  them  (e.g., 
"Here  is  a framis")  the  high  cost  of  producing  these  illus- 
trations is  not  being  recaptured  by  making  effective  use 
of  them.  Likewise,  check  to  see  if  expensive  drawings, 
attachments,  and  recordings  are  being  used  more  than  once. 

Can  a drawing  that  costs  $250  to  create  and  reproduce  be 
justified  if  students  view  it  for  only  a few  seconds? 

If  you  can  complete  a lesson  by  successively  trying  each 
multiple  choice  answer  or  by  entering  nonsense  answers  until 
you  get  the  correct  answer,  the  quality  and  validity  of  the 
lesson  should  be  examined.  Clearly,  a student's  completion 
of  such  a lesson  indicates  little  unless  the  lesson  is 
followed  by  a comprehensive  test.  For  most  students,  lessons 
with  "paths  of  least  resistance"  such  as  those  outlined  above 
produce  robot-like  behavior  but  little  learning. 
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If  si-udent  response  and  area  summary  data  are  not  examined  | 

within  a week  or  two  after  they  are  collected,  and  are  merely 
stored  on-line  or  printed  out  and  stacked  for  later  revision, 
there  are  serious  questions  about  the  effectiveness  and 
efficiency  of  lesson  revision.  It  is  a simple  matter  to  ask 
an  author  a few  questions  shortly  after  students  have  used  a 
lesson  he  wrote.  It  is  appropriate  to  ask  to  see  where  the 
data  is,  what  analyses  liave  been  made,  and  what  changes  have 
been  incorporated  into  the  lesson.  Other  useful  inquiries 
are,  "How  long  does  it  take  to  look  through  that  data  on- 
line?", "Do  you  analyze  each  student  run?",  and  "What  changes 
have  you  made  on  the  basis  of  student  response  data?" 

A student  notes  or  comments  file  is  normally  attached  to  a 
lesson  or  to  a course  in  order  to  gather  student  sug,gestions 
about  a lesson.  Reviewing  these  comments  can  be  insightful, 
though  all  comments  cannot  be  taken  at  face  value.  If  some 
of  the  suggestions  have  basis  or  merit,  a discussion  of  the 
student  views  with  the  author  may  indicate  the  author's 
viev/  toward  students.  The  presence  of  authors  who  ignore 
student  comments  out  of  hand  indicates  significant  problems. 

By  asking  an  author  about  the  topic  and  status  of  lessons 
written  by  his  colleagues,  you  may  be  able  to  simultaneously 
make  two  observations.  First,  the  author's  answer  indicates 
the  level  of  involvement  he  has  with  others,  e.g.,  if  the 
author  has  become  a hermit.  Second,  it  indicates  the  level 
of  inter-colleague  reviewing.  Note  that  the  question  should 
not  force  the  author  to  render  an  opinion  about  the  qual ity 
of  the  rest  of  the  staff,  just  what  they  are  doing.  Also,  it 
may  be  useful  to  ask  the  author  if  he  has  reviewed  any 
partially-completed  lessons.  Authors  who  are  requesting 
reviews  during  (rather  than  after)  development  should  be 
reinforced  for  this  action. 
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If  several  months  after  training,  an  author  cannot  explain 
why  he  is  writing  objectives  or  documenting  the  lesson,  fur- 
ther training  and  guidance  are  needed.  Likewise,  if  an  author 
cannot  show  the  orderly  relationship  between  the  objectives, 
the  test,  and  the  body  of  his  lesson,  the  manager  must  reluc- 
tantly conclude  that  the  instructional  design  training  hasn't 
been  understood  or  accepted. 

The  number  of  students  who  can  be  handled  by  a single  proctor 
is  a good  indication  of  the  degree  of  polish  of  the  lessons. 
Though  "polished"  lessons  are  not  necessarily  good  lessons, 
any  lessons  which  have  been  used  by  more  than  50  students  and 
I which  still  require  heavy  nroctoring  (fewer  than  12  students 

per  proctor)  are  probably  not  getting  the  attention  they 
n eed  . 

There  is  a set  of  commonly-used  computer  routines  on  the 
PLATO  system  in  what  is  called  the  "library."  If  authors  of  3 
to  6 months  of  experience  don't  know  of  the  existence  or  the 
contents  of  the  "library,"  they  may  be  spending  time  re- 
inventing routines  that  are  already  available.  A review  of 
on-line  resources  is  appropriate. 

A list  of  all  the  lessons  being  developed  or  in  use  is  found 
on  the  PLATO  system  in  the  "catalog"  lessons.  To  be  sure 
that  authors  are  not  ignoring  already-available  courseware, 
a manager  should  ask  the  author  about  other  materials  avail- 
able in  the  subject  area,  the  relationship  of  those  materials 
to  lessons  currently  being  developed  by  the  author,  and  the 
feasibility  of  incorporating  segments  of  already-written 
lessons  into  new  materials. 
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MISINTERPRETATIONS 

Two  misinterpretations  have  been  observed  often  enough 
to  warrant  warning  site  managers  about  them. 

Some  authors  misinterpreted  the  zeal  with  which  CERL 
instructional  design  consultants  promoted  data  collection, 
use  of  answer- judging  rather  than  multiple  choice,  and  inclu- 
sion of  feedback  for  specific  wrong  answers.  They  felt  these 
things  were  being  promoted  by  CERL  staff  because  CERL  wanted 
powerful  features  of  the  system  exploited  for  organizational 
glory.  In  fact  the  CERL  staff  felt  that  these  were  funda- 
mental features  (not  esoteric  ones)  and  that  if  such  features 
were  available,  but  unused,  there  was  little  justification  or 
need  for  employing  a medium  as  expensive  as  CBE.  Only 
through  use  of  the  powerful  software  features  can  one  justify 
using  CBE. 

Authors  misinterpreted  reviewers'  critiques  of  the 
positioning  of  text,  arrows,  and  feedback  as  well  as  their 
implementation  of  touch  panel  input.  Authors  tended  to 
regard  these  issues  as  unimportant  esthetic  considerations. 
They  felt  that  so  long  as  the  subject  matter  was  present  and 
accurate,  there  was  little  need  for  further  revision.  In 
fact,  CERL  and  other  groups  have  accumulated  substantial 
evidence  to  indicate  that  one  can  efficiently  "hide"  infor- 
mation on  the  screen  by  placing  it  incorrectly  and  that 
casual  implementation  of  features  such  as  touch  can  signifi- 
cantly reduce  test  scores  (Avner,  personal  communication; 
Venezky,  R.,  Bernard,  L.,  Chicone,  S. , & Leslie,  R.,  1975). 


POTFNTIALLY  COST-FFFECTIVE  USES  OF  CBE  AND  PLATO 


A CBE  system  itself  is  neither  cost-effective  nor  non- 
cost-effective.  It  is  only  the  uses  to  which  the  system  is 
put  that  determine  cost-effectiveness.  To  guide  new  sites 
and  projects  towards  uses  of  the  CBE  system  which  are  more 
likely  than  average  to  be  cost-effective,  we  have  prepared 
the  suggestions  below. 

Simulations  of  expensive  training  equipment  (including  hard- 
ware simulators)  have  been  found  to  be  cost-effective 
(Crawford,  Hurlock,  Padilla  A Sassano,  1976). 

It  is  a mistake  to  discard  all  conventional  teaching  materials 
and  replace  them  all  with  CBE  lessons.  Often  workbooks,  film- 
strips, video  tapes,  or  programmed  texts  can  be  supplemented 
with  CEE.  By  supplementing,  high  development  costs  are 
avoided,  the  high  cost  of  equipment  and  materials  continues 
to  be  amortized,  and  the  rate  of  preparation  and  validation 
of  the  improved  instruction  is  increased.  For  example,  if  a 
video-taped  training  movie  has  become  incomplete  and  obsolete 
after  three  years  (or  if  students  don't  seem  to  completely 
comprehend  it),  the  tape  could  be  followed  with  a short  CBE 
lesson  which  covers  a bit  of  new  material  and  corrects  some 
obsolete  nomenclature.  Then  an  interactive  quiz  over  both 
the  tape  and  the  CBE  supplement  could  be  administered.  Even 
if  some  comparatively  expensive  CBE  animations  must  be  pro- 
grammed, that  may  be  cheaper  than  re-shooting  the  entire 
video  tape.  Because  many  "home-made"  training  movies  forget 
to  point  out  a mechanism,  note  an  exception,  or  focus 
attention  at  the  right  time  and  place,  they  sometimes  fall 
short  of  their  potential.  A small  investment  in  supplemental 
CBE  training  might  thus  yield  large  instructional  benefits. 


Because  of  the  PLATO  system's  high  flexibility,  it  may  be 
advantageous  to  design  other  computer  teaching  systems  or 
instructional  materials  on  PLATO,  even  if  the  final  product 
is  to  be  delivered  by  a less  sophisticated  computer  or  medi- 
um. For  example,  the  evaluation  and  validation  of  instruc- 
tional materials  (including  tests)  may  require  several  itera- 
tions of  product  development,  the  gathering  and  analysis 
of  substantial  data,  and  the  preparation  of  detailed  docu- 
mentation and  reports.  It  may  be  cost-effective  to  do  all 
the  development  work  on  a PLATO  system,  then  print  the  mater- 
ials in  final,  validated  form  as  conventional  or  programmed 
texts.  Similarly,  a CMI  system  capable  of  being  implemented 
on  a mini-computer  might  be  better  designed  on  a powerful 
computer  such  as  the  PLATO  system,  then  converted  for  the 
smaller,  cheaper  computer.  By  using  the  PLATO  system  initial- 
ly, some  options  which  would  be  difficult  to  install  on  the 
mini-computer  could  be  implemented:  only  those  options  which 
were  found  to  have  great  value  would  ultimately  be  installed 
on  the  mini-computer. 

When  group  pacing  is  being  used  with  conventional  materials 
and  media,  it  may  be  cost  effective  to  provide  individualized 
remedial  training  for  slow  members  of  the  group  during  evening 
or  free  hours.  Since  in  group  pacing  the  speed  of  the  group 
is  set  by  the  slowest  member,  a relatively  high  cost  for  a 
few  students  can  be  offset  by  the  savings  of  several  hours  by 
all  members  of  the  class. 

Similarly,  when  costs  for  drop-outs  are  high,  remedial  courses 
via  CBE  may  be  cost-effective.  In  some  circumstances  it  may 
be  possible  to  combine  two  courses  teaching  the  same  topic 
to  students  with  different  backgrounds  (e.g.,  with  end  without 
a strong  math  background)  by  providing  an  individualized 
supplement  for  some  of  the  students. 
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There  are  two  approaches  for  analyzing  the  question:  "What 
size  course  is  optimal  for  CBE  lesson  implementation?"  By 
focusing  on  development  costs,  one  can  show  that  a course 
which  has  many  students  can  more  quickly  amortize  these 
costs.  By  focusing  on  delivery  costs,  one  can  demonstrate 
that  instructor  costs  for  small  classes  are  very  high  and  an 
easy  target  for  cost  effective  CBE  lessons.  Furthermore, 
small  classes  are  typically  taught  infrequently,  thereby 
inconveniencing  students  and  perhaps  lowering  total  enrollment. 
Once  developed,  a low  enrollment,  self-instructional  class 
might  be  quite  inexpensive  to  deliver  and  may  provide  high 
royalties  if  other  small  institutions  wish  to  offer  the 
course  to  their  students.  If  cost  effectiveness  is  to  be 
achieved  via  time  savings,  one  should  choose  not  only  a 
course  where  substantial  time  can  be  saved,  but  also  a course 
whose  students  are  being  well-paid  for  their  time.  For  exam- 
ple, the  money  earned  by  saving  one  hour  of  an  executive's 
time  probably  pays  for  more  hours  of  C3F  time  than  does 
saving  one  hour  of  a clerk's  time. 

In  the  situation  of  the  impending  loss  of  an  irreplaceable 
instructor,  CBE  may  be  able  to  catch  his  best  moments,  most 
thought-provoking  problems,  and  most  efficient  teaching  strat- 
egies--and  do  so  far  more  effectively  than  any  other  medium. 

Travel  expenses  for  continuing  education,  updates  on  new 
technology,  etc.  can  be  reduced  or  eliminated  by  implementing 
on-site  continuing  education  (or  refresher)  courses  via  CBE. 

In  some  professions,  however,  trips  to  take  refresher  courses 
are  treated  more  as  vacations  than  as  training  sessions.  In 
such  cases,  a CBE  replacement  for  these  trips  will  not  be 
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Some  testing  situations  require  the  simultaneous  testing  of 
a team  of  students  (e.g.,  replace  a truck  motor).  Because 
above  average  students  may  compensate  for  weaker  students, 
individual  competencies  and  weaknesses  are  difficult  to  find, 
reward,  or  correct.  Such  performance  tests  are  typically 
long  and  expensive  to  administer  and  difficult  to  grade. 

CBE  simulations  offer  a way  to  test  for  each  student  the 
cognitive  portions  of  the  task  (e.g.,  order  of  procedures), 
with  less  cuing  and  prompting  present  than  with  other  media. 

Training  in  safety  and  safe-practices  is  often  done  perfunc- 
torily, with  no  enthusiasm  and  little  effect  in  many  indus- 
tries. Unions  and  government  bodies  require  monthly  safety 
meetings  which  are  sometimes  more  of  a "rest  period"  than  a 
learning  experience.  CBE  lessons  offer  the  possibility  of 
simulations  for  procedural  tasks  which  include  "built-in," 
random  calamities  to  be  dealt  with.  Accident  prevention  can 
result  if  the  training  can  be  shown  to  have  a measurable 
impact  on  daily  procedures  and  personal  behavior. 

Calculations  about  cost-effectiveness  must  include  all  the 
constraints  and  conditions.  A good  example  of  how  an  over- 
simplified comparison  might  produce  misleading  results  is 
found  in  Lackman  (1975).  In  Lackman’s  study,  medical  simula- 
tions were  prepared  in  both  PLATO  CBE  format  and  as  paper- 
and-pencil  exercises.  When  one  calculates  the  development 
costs  for  the  simulations  in  each  medium,  one  finds  the  CBE 
format  costs  4.7  times  more  and  thus  appears  to  be  the  most 
expensive.  However,  the  simulations  were  used  by  practicing 
M.D.'s  at  conventions,  professional  meetings,  and  informal 
cour ses--always  with  voluntary  participation.  The  actual  use 
of  the  PLATO  CBE  simulations  was  found  to  be  9 times  greater 
than  that  for  paper-and-penc il  simulations.  Hence,  on  a "per 
use"  basis,  the  development  costs  were  only  half  as  much.  In 
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a similar  fashion,  one  must  carefully  consider  how  best  to 
calculate  cost/hour  for  CBE  materials.  If  one  spends  time  to 
improve  a lesson  by  allowing  students  to  complete  the  lesson 
more  quickly,  the  change  makes  the  cost/hour  less  favorable 
even  though  the  lesson  is  better. 

When  considering  the  cost  effectiveness  of  CBE  compared  to 
other  media  or  alternatives,  consider  what  options  are  actu- 
ally available.  For  example,  it  may  be  cheaper  to  use  another 
computer  to  do  some  numerical  integrations,  but  if  the  other 
computer  is  blocks  away  or  offers  only  overnight  service,  a 
CBE  system  with  higher  costs,  but  faster  turnaround  might  be 
preferable.  Likewise,  video  tape  might  be  the  most  appropri- 
ate medium  for  a topic,  but  if  you  lack  video  equipment,  CPE 
animations  might  be  the  most  cost-effective  option  available. 

It  may  be  more  cost-effective  to  write  new  courses  using  CRE 
than  to  convert  existing  courses.  This  is  particularly  true 
when  substantial  capital  costs  for  equipment,  texts,  rooms, 
etc.  have  already  been  made.  Even  if  a CPF-simulated  lab 
is  cheaper  and  teaches  better  than  actual  lab  experience,  the 
fact  that  the  cost  of  existing  lab  equipment  and  special  class- 
rooms has  not  yet  been  amortized  may  mean  that  switching  to 
CBE  simulations  is  impractical. 

Much  of  the  time  savings  arrived  at  by  converting  a conven- 
tional lock-step,  lecture-format  class  to  one  which  is  com- 
puter-based are  due  to  time  saved  through  self-pacing.  Some 
opponents  of  CBE  argue  that  only  the  savings  of  CBE  over  and 
above  that  for  other  self-paced  media  can  be  considered  when 
calculating  cost-effectiveness.  However,  the  management 
function  of  CBE  is  often  the  key  to  being  able  implement 
and  operate  a self-paced  system.  That  is,  without  CBE’s 
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management  capability,  it  would,  not  be  feasible  in  some 
cases  to  design  a self-paced  course  using  other  cheaper 
media. 
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WHY  USE  A CBE  SYSTI  LIKE  PLATO? 

The  decision  to  use  CBE,  and  the  PLATO  system  in  partic- 
ular, may  already  have  been  made  when  the  site  manager  assumes 
his  duties.  Nevertheless,  it  may  be  prudent  for  the  manager 
to  review  the  decision.  Below  is  a brief  list  of  character- 
istics of  the  PLATO  system  which  make  it  worth  using. 

It  is  a computer  and  can  do  things  computers  do. 

For  example,  it  can  store  and  process  data.  On 
the  other  hand,  it  is  not  a general-purpose  com- 
puter; it  has  restricted  database  operations  and 
relatively  slow  input  and  output  from  terminals. 

It  has  a uniquely  powerful  language  for  teaching  and 
performing  other  special  ^unctions. 

TUTOR  is  widely  recognized  and  imitated  because 
of  its  powerful  features.  Moreover,  it  continues 
to  add  new,  useful  features.  However,  TUTOR 
programs  are  not  readily  transferable  to  other 
computers . 

The  PLATO  system  is  equipped  with  a unique  graphics 
terminal  as  an  output  device. 

The  terminal  and  the  auxiliary  devices  which  can 
be  attached  to  it  offer  a broad  potential  for 
nearly  any  important  kind  of  interaction  with  a 
user.  However,  unless  the  training  requires 
these  features  and  the  lesson  developers  make 
use  of  them,  the  potential  is  wasted. 

The  communications  network  provided  by  the  PLATO  system 
is  unmatched  in  terms  of  the  breadth  of  types  of  users 
and  types  of  information  communicated. 

Lesson  writers  can  not  only  share  courseware,  but 
can  co-develop  lessons,  review  lessons  during 
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development,  etc.  Nevertheless,  there  continues 
to  be  duplication  of  effort  and  missed  opportuni- 
ties for  inter-site  communication. 

The  ability  of  the  PLATO  terminal  to  control  external 
devices  makes  it  ideal  for  running  psychology/education 
experiments . 

Student  interaction  data  related  to  educational 
goals  can  be  gathered  rather  automatically. 

Timing,  presentation  of  graphic  and  other  stimuli 
can  be  interactively  controlled. 

Both  the  hardware  and  software  are  designed  so  that  a 
student  can  begin  to  use  the  terminal  after  2-3  minutes 
of  orientation.  No  computer  jargon,  programming  skills, 
or  technical  abilities  are  required. 

One  can  find  somewhere  on  another  computer  or  in  some 
other  medium  most  of  the  features  cited  above  (with  some 
exceptions,  such  as  the  TUTOR  language).  If  only  one  of  the 
features  is  needed,  it  can  probably  be  obtained  more  readily 
or  inexpensively  elsewhere  than  the  PLATO  system.  However, 
one  of  the  great  virtues  of  the  PLATO  system  is  the  fact 
that  many  useful  facilities  are  available  from  and  integrated 
into  a single  computer  system. 

Some  examples  of  these  fruitful  combinations: 

Computer-management  of  CAI  and  off-line  materials. 
Realistic  CBE  simulations  via  graphics,  database,  and 
interterminal  communication  features  and  terminal 
peripherals. 

Powerful  teaching  strategies  involving  sophisticated 

branching  algorithms,  drill  paradigms,  computerized 
diagnosis  of  student  weaknesses,  and  student  control 
of  sequencing. 
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On-line  conferencing  with  immediate  (telephone-like)  or 
delayed  (letter-like)  response  among  large 
populat ions . 


L 
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CONTINGENCY  PLANNING 

Few  laws  are  so  invariant  and  oft-quoted  as  Murphy's: 

"If  anything  can  go  wrong,  it  will."  Realizing  the  wide 
applicability  of  this  law,  it  is  only  reasonable  to  construct 
contingency  plans  for  some  of  the  predictable  crises  in  the 
life  of  a CBE  site.  Below  is  a list  of  likely  problems  and 
plausible  alternatives. 

The  system  or  local  site  experiences  a "down"  period  longer 
than  one  class  period--as  long  as  several  days. 

Teach  conventionally  in  lecture  format  using  lesson 
authors  as  instructors. 

Use  hardcopy  version  of  CBE  lessons  prepared  before  the 
downtime  period. 

Postpone  training. 

Lesson  production  goals  are  not  met,  or  substantial  loss  of 
experienced  personnel  occurs. 

If  available,  hire  expensive  experienced  staff. 

If  time  permits,  train  new  staff. 

Reduce  the  scope  of  the  project. 

Extend  the  term  of  the  project. 

Hire  outside  contractors  to  complete  the  development. 

The  student  enrollment  in  target  courses  becomes  so  low  that 
evaluation  statistics  will  not  be  reliable. 

Extend  the  project  to  increase  the  sample  size. 

If  you  are  dividing  the  students  into  experimental  and 
control  groups,  make  them  all  experimental  and  use 
previous  classes  for  control. 

Modify  the  materials  to  teach  a high-flow  class. 
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The  student  enrollment  becomes  so  large  that  there  are  too 
many  students  to  schedule,  or  access  to  terminal  time  is 
reduced  so  that  sufficient  terminal  time  is  not  available. 

Reduce  the  amount  of  authoring  and  the  number  of 
terminals  devoted  to  authoring. 

Schedule  authoring  for  weekends  and  evenings. 

Teach  part  of  the  students  conventionally. 

Allow  students  to  self-schedule  themselves  during 
non-prime  time.  J 

i 

] 

There  is  a persistent  problem  with  obtaining  sufficient  ' 

computer  storage  (ECS)  for  all  student  lessons. 

Schedule  classes  so  that  students  are  likely  to  be 
sharing  a lesson  in  ECS.  Essentially  this  means 
reducing  the  self-pace  mode  and  adopting  a more  lock- 
step  orientation. 

If  practical,  invest  the  time  of  programmers  to  divide 
very  large  lessons. 

The  funding  or  ability  to  perform  the  project  is  suddenly 
terminated  . 

Prepare  hardcopy  versions  of  all  completed  instruction. 

Re-program  the  materials  for  other  computers  or  rewrite 
the  materials  for  other  media. 

Although  none  of  the  potential  alternatives  is  supplied 
with  a caveat,  each  should  be!  Note  also  that  this  list  is 
far  from  complete.  One  of  the  main  purposes  of  these  guide- 
lines is  to  present  examples  of  situations,  problems,  and 
alternatives.  A more  complete  list  of  contingencies  would 
contain  options  for  problems  with  poor  lesson  quality,  person- 
nel problems,  skeptical  traditional  instructors,  etc.  The 
manager  of  each  site  should  prepare  his  own  list,  and  if 
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necessary,  pet  it  approved.  Such  advanced  planning  allows 
for  thoughtful  procedures  such  as  the  "backwards  implementa- 
tion" discussed  in  part  I. 
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GUIDELINES  FOR  CONSTRUCTING  PROJECT  TIMELINES 

One  of  the  most  important  problems  encountered  in 
planning  and  managing  a CBE  site  is  that  of  estimating  the 
size  of  the  staff  and  the  total  time  needed  to  complete  a 
major  lesson  development  effort.  Sites  frequently  over- 
estimate their  production  efficiency  or  fail  to  take  account 
of  non-production  tasks  and  are  thus  unable  to  complete  the 
intended  project.  For  that  reason  these  guidelines  have 
already  recommended  that  sites  allow  for  "soft"  failures  or 
ways  to  demonstrate  success  despite  not  achieving  all 
original  goals. 

Although  "soft"  failures  are  better  than  "hard"  ones, 
what  is  needed  is  a better  basis  for  determining  the  magni- 
tude of  the  task  and  the  productivity  of  the  staff.  The 
ability  to  make  accurate  forecasts  will  lower  the  frequency 
and  severity  of  mid-project  adjustments  and  reduce  the 
uncertainties  of  CBE  courseware  development. 

The  following  discussion  assumes  that  CBE  is  new  to 
nearly  all  staff  members  at  a site.  (Jfter  a site's  staff 
members  have  experience,  they  can  make  estimates  by  extrap- 
olation, but  during  the  planning  stages  of  a site,  no  lessons 
have  been  written.  Yet  this  is  the  period  when  the  best 
estimates  are  needed.) 

In  spite  of  the  fact  that  local  situations  create  issues 
which  cannot  be  answered  by  generalizations,  we  feel  that 
we  have  gleaned  some  insight  which  can  be  used  in  project 
planning  and  management.  In  order  to  make  the  problems  and 
constraints  as  realistic  as  possible,  we  have  prepared  the 
following  example  of  how  NOT  to  use  estimates  of  lesson 
production  time  to  plan  a project. 


f 
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Project  History  of  MNL 

The  NeverNeverLand  project  staff  read  several  reports 
which  indicated  that  groups  similar  to  theirs  had  spent  250 
author  hours  to  write  each  hour  of  lesson  material.  The  MNL 
project  wished  to  create  100  hours  of  CBE!  material  and  calcu- 
lated an  author  working  2000  hours  per  year  could  develop 
eight  lessons.  They  thus  requested  and  received  authoriza- 
tion to  hire  13  staff  for  two  years.  They  were  told  that  if 
the  evaluation  results  were  good,  the  project  would  be  con- 
tinued. All  the  authors,  programmers,  illustrators,  and 
technicians  were  working  full  time  within  three  months  after 
the  project  started,  and  within  six  months  they  had  learned 
enough  of  the  authoring  language  to  be  producing  lessons. 

The  fact  that  the  last  three  months  of  the  project  were 
set  aside  for  evaluation  and  could  not  actually  be  used  for 
development  did  not  bother  the  project  director  because  the 
rate  of  lesson  production  was  faster  than  the  assumed  rate. 
During  the  second  quarter  of  the  first  year  of  the  project, 
the  average  production  time  was  250  hours  per  hour;  the 
third  quarter,  it  was  down  to  150  hours  per  hour!  Though 
the  fully-operat ional  lesson  development  period  was  to  be 
only  15  months  (24  months  minus  6 months  arriving  and 
getting  trained  minus  3 months  evaluation),  the  production 
rate  achieved  at  NNL  seemed  destined  to  be  twice  as  fast  as 
had  been  predicted. 

The  project  director  was  surprised  when  the  fourth 
quarter  production  rate  rose  to  175  hours  per  hour  rather 
than  continuing  to  decline  as  he  thought  it  would.  Feeling 
his  authors  had  become  a little  too  relaxed,  he  put  on  the 
pressure  to  produce.  During  the  fifth  quarter,  it  took  an 
average  of  150  hours  to  produce  each  one  hour  lesson,  but  two 
authors  disliked  the  pressure  and  quit.  With  only  six  months 
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left,  hiring  untrained  staff  was  out  of  the  question;  instead 
one  experienced,  but  expensive,  author  was  hired  to  replace 
the  two  who  had  quit. 

Before  the  final  evaluation  period  began,  several  prob- 
lems reared.  The  NNL  project  director's  boss  asked  to  see 
some  evaluation  data  to  show  department  administrators  so 
that  the  project  could  continue  to  be  funded.  The  boss 
understood  that  the  evaluation  period  had  not  begun,  but  he 
needed  some  evidence  that  the  CBE  system  should  be  kept,  and 
he  needed  it  before  the  end  of  the  current  contract.  He 
would  even  be  happy  with  lesson  validation  data,  he  said. 

Unfortunately,  lesson  validation  data  was  in  short 
supply.  The  project  director  admitted  that  the  lessons  used 
during  the  evaluation  period  would  include  some  which  had  not 
been  validated  prior  to  the  evaluation  period  (i.e.,  they  had 
not  yet  been  shown  to  teach  effectively)  and  some  lessons 
which  were  not  even  written  yet  but  which  hopefully  would  be 
finished  during  the  evaluation  period.  The  difficulty  was 
caused  by  the  fact  that  only  three  classes  a year  were  sched- 
uled (Fall,  Spring,  and  Summer).  Most  lessons  were  revised 
twice  before  they  were  validated  and  hence  few  lessons 
written  during  the  second  year  had  had  enough  use  to  have 
reached  their  validation  criterion.  Most  of  the  validated 
lessons  were  those  written  the  first  year,  when  production 
was  slow  and  experience  was  minimal. 

Lacking  sufficient  data  on  which  to  base  a decision,  the 
department  was  forced  (rather  unhappily)  to  grant  the  project 
an  extension  for  only  a few  months,  until  the  evaluation 
phase  could  be  completed.  The  department  was  willing  to  con- 
tinue the  project,  but  needed  evidence  to  justify  that  decis- 
ion to  a budget-conscious  administration. 

During  this  extension  period,  the  NNL  staff  divided 
their  time  between  running  the  evaluation  and  finding  new 
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jobs  in  more  stable  environments.  Though  the  evaluation 
results  were  seeminply  positive,  only  one  class  of  students 
was  included  in  the  evaluation  sample  (because  only  three 
classes  started  a year  and  only  one  could  be  run  in  the  three 
months  set  aside  for  evaluation).  With  such  a small  sample, 
statistical  tests  had  only  limited  power  to  detect  differences 
in  student  performance.  Furthermore,  some  skeptics  said  that 
summer  school  students  were  so  atypical  that  the  results 
would  not  be  applicable  to  the  students  in  the  other  sessions. 

Faced  with  a depleted  staff,  positive  evaluation  results, 
and  another  long  training  period  for  new  staff,  the  depart- 
ment funded  CBF  support  for  continued  student  operations  and 
lesson  maintenance,  but  no  more  new  development.  Some 
members  of  the  administration  felt  that  the  department  had 
been  used  as  a training  ground  for  CBE  authors,  who  left  for 
better  jobs  after  being  trained.  Though  they  acknowledged 
the  student  satisfaction  and  the  enhanced  performance,  they 
discouraged  any  new  lesson  development. 

The  reader  should  be  advised  that  although  there  is  no 
MeverNeverLand  site,  the  details  of  this  fictional  account 
are  a composite  of  events  from  the  case  histories  of  several 
actual  sites. 

Let  us  examine  the  errors  and  misconceptions  of  the  NNL 
staff  in  planning  their  project. 

Post  mortem  Analysis 

The  story  ended  in  the  place  where  the  planning  should 
have  started:  the  evaluation.  After  choosing  objectives, 
one  should  begin  immediately  to  decide  how  to  evaluate  the 
project.  Using  "power  analysis"  techniques  (Cohen,  1969), 
one  can  determine  the  minimum  number  of  subjects  one  needs 
during  the  summative  evaluation  period  to  have  a reasonable 


J 

I 


,1 


132 


I 

<» 

chance  of  detecting  performance  differences.  That  informa- 
tion, combined  with  average  class  sizes,  can  be  used  to  deter- 
mine the  length  of  the  evaluation  period.  As  NNL  found  out, 
the  need  for  evaluation  data  comes  before  the  last  day  of  the 
project.  Because  final  evaluations  are  usually  not  completed 
before  the  end  of  the  project,  it  is  necessary  to  collect  and 
interpret  some  preliminary  data.  That  can  come  from  the 

formative  development  or  validation  data  or,  if  several  ' 

classes  of  students  participate  in  the  evaluation  period,  I 

from  data  from  an  early  class.  i 

Assume  that  power  analysis  indicates  that  100  students 
are  needed  for  the  evaluation.  If  classes  average  40  stu-  ■! 

dents  and  are  taught  three  times  per  year,  the  entire  final  j 

year  of  the  project  will  be  needed  to  gather  summative  data.  j 

If  the  lessons  require  two  revisions  before  validation,  three  I 

groups  of  students  must  use  each  lesson,  thus  adding  perhaps  | 

( 

another  year  to  the  project.  (Because  initial  testing  and 

formative  evaluation  can  be  conducted  with  students  from  | 

other  courses  in  many  cases,  the  one-year  validation  period  J 

can  often  be  shortened.)  ^ 

At  the  beginning  of  a project,  staff  are  learning  about  j 

the  language  and  CBE;  lesson  development  may  be  slow  for  4 to  6 
months.  Using  the  assumptions  above,  one  finds  nearly  30 
months  are  needed  for  training,  validation,  and  evaluation 
alone.  Development  time  must  be  added  to  the  30  month  sum. 

Unless  the  validation  or  evaluation  periods  can  be  shortened, 
such  a project  must  be  more  than  three  years  in  length. 

Projects  which  attempt  to  perform  summative  evaluation 
simultaneously  with  lesson  validation  have  the  following 
problems.  Since  the  lessons  are  changing  during  the  evalua- 
tion period,  the  summative  data  collected  are  not  comparable. 

Furthermore,  problems  with  one  lesson  affect  other  lessons, 

making  detection  and  location  of  pedagogical  stumbling  blocks  ' 

more  difficult  than  if  the  lessons  were  validated  individually. 
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Although  not  all  endeavors  require  three  years,  it  is 
the  failure  to  plan  projects  in  detail  or  the  treatment  of 
development  as  merely  so  many  man-months  of  activity  which 
causes  many  of  the  difficulties  experienced  by  CBE  sites. 

It  should  also  be  noted  that  longer  projects  with  fewer 
people  may  be  more  efficient.  Staff  members  on  long  pro- 
jects spend  a smaller  proportion  of  their  time  in  training 
and  can  take  advantage  of  the  experience  they  gain.  Major 
CBE  projects  which  begin  with  inexperienced  staff  should 
not  be  shorter  than  three  years.  It  is  also  important  to 
choose  a course  that  is  taught  several  times  a year;  authors 
must  have  adequate  supplies  of  student  "guinea  pigs"  to  test 
their  lessons. 

Where  effectiveness  of  the  medium  is  being  evaluated, 
it  is  often  best  to  limit  the  total  quantity  of  material 
being  generated  to  a minimum  and  to  target  this  material 
for  applications  with  high  usage.  This  insures  adequate 
time  to  experiment  with  several  alternative  ways  of  using 
the  new  medium  while  maximizing  chances  for  student  testing 
of  the  draft  materials.  In  too  many  "evaluations,"  an 
overly  ambitious  production  plan  has  converted  an  evaluation 
of  a medium  or  technology  into  an  evaluation  of  the  ability 
of  the  instructional  design  staff  to  produce,  on  their  first 
try,  perfect  material  in  a totally  unfamiliar  med  .urn  while 
under  severe  stress! 

Production  Rates 


What  rate  of  production  should  be  expected?  Group 
rates  of  75  hours  to  600  hours  and  individual  rates  of  17 
hours  to  2000  hours  have  been  reported  formally  and  informal- 
ly. The  range  is  wide,  not  only  because  of  variations  in 
the  difficulty  of  the  task  and  the  ability  of  the  authors, 
but  also  because  of  differences  in  what  "writing  a lesson" 
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includes.  Most  staff  who  prepare  lessons  also  spend  time 
revising  lessons,  giving  demonstrations,  using  the  lessons 
to  teach,  reviewing  lessons  of  other  authors,  constructing 
routers,  aiding  the  evaluation,  and  training  new  staff.  F'alf 
or  more  of  the  time  of  staff  may  thus  be  spent  in  duties  not 
directly  related  to  lesson  production.  That  is  why  the  NML 
seemed  to  be  doing  so  well.  A rate  of  250  hours  may  have 
characterized  the  entire  MHL  project  effort  (e.g.,  the  total 
time  consumed  by  each  author  during  all  stages  of  the  pro- 
ject divided  by  the  output  of  the  project).  Production  of 
first  draft  lessons  generally  takes  less  than  that  ?50  hours 
per  hour  and  certain  individual  lessons  often  can  be  pro- 
grammed in  very  much  less  time.  A further  source  of  confusion 
in  reporting  production  rates  is  deciding  which  author-support 
time  to  include.  The  time  of  programmers  is  certainly  includ- 
ed, but  should  time  for  proctors,  secretaries,  illustrators, 
and  hardware  repairmen  also  be  counted? 

Courseware  production  times  can  be  misleading  for 
several  reasons.  When  starting  with  an  inexperienced  staff, 
a project  director  finds  that  the  "learning  curve"  reduces 
production  times  quickly.  The  first  lesson  each  author 
writes  may  take  600  hours  to  produce,  the  second  300,  the 
third  200.  Furthermore,  each  lesson  tends  to  improve  in 
quality.  The  production  time  seems  destined  to  continue 
decreasing.  Instead,  three  factors  tend  to  make  the  produc- 
tion time  level  off  or  even  increase. 

A major  burden  for  authors  mid-way  through  a project  is 
the  increasing  quantity  of  data  gathering,  interpretation,  and 
revision  activities  required.  Initially,  an  author  has  only 
a few  lessons  to  revise;  he  can  spend  most  of  his  time  creat- 
ing new  lessons.  There  may  be  only  a few  periods  when  there 
are  students  at  an  appropriate  place  in  their  course  to  use 
his  lessons.  After  a year  or  so,  the  author  must  devote  a 
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considerable  amount  of  his  time  to  revisinR  and  maintaining 
existing  lessons  and  testing  new  ones. 

A second  factor  tending  to  increase  courseware  produc- 
tion time  is  the  finding  that  the  "learning  curve"  typically 
shows  "setbacks"  (Avner,  personal  communication)  during  which 
a new  author  may  drop  back  to  low  production  rates.  Avner 
discovered  that  these  plateaus  usually  happened  when  the 
author  was  perfecting  a new  instructional  technique.  Each 
new  technique  added  to  the  repertoire  of  the  new  author  would 
show  a "learning  curve"  of  production  efficiency  as  it  was 
applied  to  more  and  more  lessons.  Until  an  adequate  reper- 
toire was  developed  for  the  specific  needs  of  the  author’s 
project,  it  was  quite  common  to  see  erratic  shifts  of  from  20 
to  600  hours  per  hour  as  an  individual  author  added  and 
exploited  each  new  technique.  In  areas  where  a wide  range  of 
techniques  are  required,  it  took  a year  or  more  to  develop  a 
repertoire  adequate  to  make  full  use  of  the  capabilities  of 
CBE.  However,  once  the  repertoire  was  present,  exceedingly 
efficient  production  of  effective  material  often  resulted. 
Taking  into  consideration  the  amount  of  time  needed  to  learn 
these  skills  and  the  amount  of  effort  often  required  of  an 
author  for  other  duties,  it  is  not  realistic  to  expect  that 
many  one-hour  lessons  will  be  produced  by  a new  author  in  17 
hours  during  a 3-year  project. 

This  need  for  training  of  users  of  a medium  should  not 
be  surprizing.  Few  would  expect  a group  of  novices  to  make 
effective  and  efficient  use  of  even  well-established  media 
such  as  film  or  videotape  without  a substantial  period  of 
learning  and  practice.  At  the  same  time,  it  is  easy  to  be 
misled  by  the  apparent  ease  with  which  a novice  is  able  to 
produce  first-draft  materials  in  almost  any  medium  (film  and 
CBE  inc luded  ) . 


As  a project  matures,  its  authors  gain  additional  re- 
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sponsibilities . When  few  lessons  have  been  written,  there 
is  very  little  need  for  routers,  for  bookkeeping  about  the 
names  and  locations  of  lessons,  and  for  scheduling  and 
proctoring  classrooms  full  of  students.  These  activities 
are  added  to  the  tasks  of  authoring,  reviewing  lessons  and 
preparing  for  the  evaluation.  These  accumulating  duties  act 
as  a third  force  to  slow  lesson  production  during  project 
maturity . 

A final  problem  suggested  by  the  NNL  example  is  that 
estimates  of  the  number  of  developers  needed  should  be  adjust 
ed  upward  to  allow  for  non-development  staff  such  as  proctors 
illustrators,  repairmen,  etc.  These  staff  may  be  needed 
only  part  time  during  limited  phases  of  the  project.  Their 
skills  are  important,  but  one  cannot  expect  to  replace  devel- 
opment staff  with  support  staff  and  still  meet  the  same  pro- 
duction goals. 

A cautionary  note  about  the  production  rate  of  250 
hours  used  in  the  NNL  example  is  necessary.  Several  projects 
have  derived  or  used  a value  of  250-300  hours  for  overall 
production  time  including  training,  revision,  etc.  We  can- 
not endorse  this  or  any  other  "rule  of  thumb"  value  because 
of  the  vast  differences  between  sites  and  projects.  In  the 
earliest  stages  of  project  planning,  it  might  be  reasonable 
to  estimate  300  hours  per  hour  of  material  to  approximate  the 
scope  of  a project.  After  the  scope  is  established,  the 
objectives  determined,  and  an  evaluation  sketched  out,  one 
should  request  several  independent  CPE  consultants  to  refine 
the  plans  and  provide  estimates  of  project  milestones,  staff 
needs,  and  timelines.  Providing  rules  for  making  such 
estimates  is  beyond  the  purview  of  these  guidelines.  The 
consultants'  estimates  must  be  verified  during  the  first  year 
of  the  project,  observing  the  caveats  noted  in  this  section 
about  the  fickleness  of  trends. 
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If  it  is  not  possible  to  obtain  the  aid  of  a qualified 
consultant,  you  should  use  very  conservative  estimates.  Avner 
Ipersonal  communication)  indicates  that,  based  on  a study  of 
27  new  authors,  one  could  be  90^^  confident  that  l'b%  of  a 
group  of  experienced  teachers  with  good  verbal  skills  (all 
were  college  graduates)  would  be  able  to  produce  an  hour  of 
material  during  their  first  year  in  about  ^30  hours  or  less 
of  effort.  This  estimate  includes  only  minimal  formative 
evaluation.  A rule-of-thumb  based  on  these  data  would  thus 
suggest  that  a project  with  a one-year  lesson  development 
phase  should  allot  no  more  than  5 hours  of  material  to  each 
new  author  during  the  year  and  tnat  4 authors  be  provided  for 
every  15  hours  of  material  to  be  produced. 
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